ROMweasel 0.9.9

[Koston]

Hello nerds,

I wrote myself a little tool for finding and downloading individual game ROMs on the MiSTer. This is mainly intended for CD based systems, when having local copies of full libraries isn't sensible. Or when my local library just happens to be missing something I very urgently must play.

I'd appreciate any feedback from users, coders etc. if you think it'd be worthwhile developing it further or how. Download from MiSTer shell:

(Note: Project moved to Github)
https://github.com/Koston-0xDEADBEEF/MiSTer-ROMweasel

Code: Select all

cd /media/fat/Scripts && curl -kLO https://github.com/Koston-0xDEADBEEF/MiSTer-ROMweasel/raw/main/romweasel.sh

There's no documentation yet, and hopefully never needs to be. This is intended to be simple enough to use, to not require a manual.

There are few instructions now on Github, based on feedback from this thread.

[venice]

A really handy tool.
Many Thx

[netbeui]

I'll definitely check this out later - thanks Koston! Is this supposed to go in the scripts folder and then you run it from within MiSTer? Or just download the file in a shell and run it from there?

[Koston]

Is this supposed to go in the scripts folder and then you run it from within MiSTer? Or just download the file in a shell and run it from there?

Either way works.

[LamerDeluxe]

This is a fantastic idea. Saves the extra steps copying it from my PC.
So far it gives me 0B size for the TGFX16 CD and PSX games I selected and then it doesn't download anything.

[mrchrister]

Works well for me, thanks! Tested PSX US and NES

[FPGA64]

Just a warning. Dont download random scripts to your mister until you have checked them out. You never know if its doing what it says till you do.

[Koston]

So far it gives me 0B size for the TGFX16 CD and PSX games I selected and then it doesn't download anything.

Did you actually select anything before trying to download? You need to select items with spacebar.

If that is the issue, it's an oversight on my behalf - I realised too late the script cannot be used at all without a keyboard. I intend to investigate how to make it usable with just a controller.

[LamerDeluxe]

So far it gives me 0B size for the TGFX16 CD and PSX games I selected and then it doesn't download anything.

Did you actually select anything before trying to download? You need to select items with spacebar.

If that is the issue, it's an oversight on my behalf - I realised too late the script cannot be used at all without a keyboard. I intend to investigate how to make it usable with just a controller.

Thanks, that worked! That part wasn't self explanatory yet.

[Neocaron]

So far it gives me 0B size for the TGFX16 CD and PSX games I selected and then it doesn't download anything.

Did you actually select anything before trying to download? You need to select items with spacebar.

If that is the issue, it's an oversight on my behalf - I realised too late the script cannot be used at all without a keyboard. I intend to investigate how to make it usable with just a controller.

I had the same issue, A working controller input or a "select" option in the lower menu would be nice.
I'd be happy with a "select all" as well. It might come in handy.

I can confirm it works well with a keyboard using space.

[Koston]

I'd be happy with a "select all" as well. It might come in handy.

This feature is currently absent on purpose.

I'm considering adding a separate code path for downloading full libraries, but only if the total library size is reasonable; I don't want to encourage wasting archive.org's bandwidth. They're struggling as it is, and their work preserving digital history is very important.

[Neocaron]

I'd be happy with a "select all" as well. It might come in handy.

This feature is currently absent on purpose.

I'm considering adding a separate code path for downloading full libraries, but only if the total library size is reasonable; I don't want to encourage wasting archive.org's bandwidth. They're struggling as it is, and their work preserving digital history is very important.

I totally get it.
Could be interesting to even have our own personal cloud library hooked to the script. I'll see if I can get that to work with onedrive

[rhester72]

I'm considering adding a separate code path for downloading full libraries, but only if the total library size is reasonable; I don't want to encourage wasting archive.org's bandwidth. They're struggling as it is, and their work preserving digital history is very important.

I wouldn't worry too much about that...people are already doing it (frequently), and archive.org throttles connections as a result. Truly large ones require authentication, so there's that.

[capitaineflam25]

Wow, this is really nice ! Thanks for sharing your work.

What is the recommanded way to configure downloads to /media/usb0/games/ instead of /media/fat/games/

[Koston]

What is the recommanded way to configure downloads to /media/usb0/games/ instead of /media/fat/games/

The script sources /media/fat/Scripts/.config/romweasel/settings.sh for user settings, or if the file doesn't exist, creates it.

[capitaineflam25]

The script sources /media/fat/Scripts/.config/romweasel/settings.sh for user settings, or if the file doesn't exist, creates it.

Thanks !

[georgemog]

This is a fantastic script which makes everything very easy to use.

For the file /media/fat/Scripts/.config/romweasel/settings.sh, I updated it but made the mistake of not removing the '#' character in the line. Once I fixed it. I was able to download files to the correct place without issues.

I did find one issue with downloading PSX files and that is that they require a unique subdirectory for each game. I have added the following lines in bold to the download_roms procedure.

# Download selected ROMs
download_roms () {
tags=(${*})
rominfo="$(get_rom_info $tags)"
rominfo+="\nDownload the selected game(s) to \"${DEST_GAMEDIR}\" ?\n"

${DIALOG} --title "Information for selected ROM(s)" --clear --cr-wrap \
--yesno "$rominfo" $(( $MAXHEIGHT / 2 )) $MAXWIDTH 2>${DIALOG_TEMPFILE}
retval=$?
[[ $retval -eq $DIALOG_CANCEL ]] && return
[[ $retval -ne $DIALOG_OK ]] && cleanup

# In case the file exists already, cURL will attempt to continue the download
curl_opts=(--connect-timeout 5 --retry 3 --retry-delay 5 -C - -kL)

# For PSX games create a subdirectory for the .chd file to sit in
if [ "$DEST_GAMEDIR" = "*PSX*" ]; then
odir=`echo $ofile | sed 's/.chd//g'`
tmp=$DEST_GAMEDIR"/"$odir
DEST_GAMEDIR=$tmp
fi

# Make sure target directory exists or if user wants it to be created
if [[ ! -d $DEST_GAMEDIR ]]; then
${DIALOG} --title "Warning" --clear --cr-wrap --yesno \
"Directory \"${DEST_GAMEDIR}\" doesn't exist.\n\nCreate it?" \
10 82 2>${DIALOG_TEMPFILE}
retval=$?
[[ $retval -eq $DIALOG_CANCEL ]] && return
[[ $retval -ne $DIALOG_OK ]] && cleanup
mkdir -p $DEST_GAMEDIR
fi
...

This seems to resolve the problem for me.

[LamerDeluxe]

I did find one issue with downloading PSX files and that is that they require a unique subdirectory for each game. I have added the following lines in bold to the download_roms procedure.

This seems to be the case with all chd files, not just for the PSX core.
Not sure about other CD formats, but it would probably be the same, as it is used to detect if a CD change needs a core reset.

[Koston]

I did find one issue with downloading PSX files and that is that they require a unique subdirectory for each game.

I was not previously aware of this! Thank you for pointing it out.

Looking at the PSX core documentation, multiple CD games only require all images in the same folder - so it's not required to have individual directories per game. But there is another, very compelling reason for doing it:

Code: Select all

Games that are in their own folder will create it's own memory card in media/fat/saves/psx as .sav

I've been cursing the cumbersome handling of save files with the PSX core, but this should get rid of the problem entirely.

I will modify your patch a bit and push an update to Github.

[LamerDeluxe]

I've been cursing the cumbersome handling of save files with the PSX core, but this should get rid of the problem entirely.

I will modify your patch a bit and push an update to Github.

Please make sure this works for all chd files, not just the PSX ones.

[Koston]

Commit 076fd2b:

Place CD images into game-specific subdirectories

CD based systems benefit from having each game in their own directory,
especially so with multi-CD games.

Implemented heuristics for determining a suitable unique path for
each game. There are some corner cases when it doesn't work, in such
case the files are simply placed to the ROM root and left to the user
to relocate as they see fit.

Also fixed 4-byte integer overflow in total ROM size calculations.

This change turned out to be a lot more complex than anticipated.

[Koston]

Also,

A big thank you to everyone who tested this! Feedback has been quite valuable.

I'm not sure how much more time I have to spend on this (just doing it for fun anyway), but here's a shortlist of features I'm looking into next:

• Enable controller-only operation DONE
• Full library download function (for reasonably sized libraries only)
• Keep session state in disk cache for automatically continuing unfinished downloads (currently user needs to manually re-select same games to continue download from where it left off)
• Re-using CHD placement logic for reorganizing already existing local files DONE

[LamerDeluxe]

Commit 076fd2b:

Place CD images into game-specific subdirectories

CD based systems benefit from having each game in their own directory,
especially so with multi-CD games.

Implemented heuristics for determining a suitable unique path for
each game. There are some corner cases when it doesn't work, in such
case the files are simply placed to the ROM root and left to the user
to relocate as they see fit.

Also fixed 4-byte integer overflow in total ROM size calculations.

This change turned out to be a lot more complex than anticipated.

Don't know if this helps as an example, but this is a (Python) script made by user @bootsector on the MiSTer Discord. I used this to correctly organize my chd files for multiple cores.

Code: Select all

import os
import sys
import re

MULTI_DISK_PATTERN = r'\(Disc \d+\)'

def gamedirname(filename):
    multidisk = re.search(MULTI_DISK_PATTERN, filename)
    if multidisk:
        return(filename[0:multidisk.start()].strip())
    else:
        return(filename[0:filename.rfind(".")].strip())

if len(sys.argv) != 2:
    sys.exit("Usage: chd2folders <CHDs dir>")

for file in os.listdir(sys.argv[1]):
    if file.lower().endswith(".chd"):
        file = os.path.join(sys.argv[1], file)
        dir = gamedirname(file)
        if not os.path.exists(dir):
            os.mkdir(dir)
        os.rename(file, os.path.join(dir, os.path.basename(file)))

[Koston]

Commit 076fd2b

This change turned out to be a lot more complex than anticipated.

Don't know if this helps as an example, but this is a (Python) script made by user @bootsector on the MiSTer Discord. I used this to correctly organize my chd files for multiple cores.

That Python script seems to be doing same as I initially did on my PSX game directory; generate directory names by simply stripping "(Disc #)" and everything after it from the filename. Adhoc:

Code: Select all

% for f in *.chd; do d=${${(Q)f%.chd}// \(Disc [0-9]\)*/} ; [[ -d $d ]] || mkdir "$d" ; mv "$f" "$d" ; done

This solution works most of the time and when it doesn't, it "fails" by only putting discs from different multi-CD game sets into the same directory.

I've tried explaining the additional logic I wrote for this in comments of get_rom_gamedir() function. Or, you can just test it yourself by filtering the game list with keyword "disc" and then seeing with "ROM Info" what kind of target directory names it comes up with.

There's certainly not much merit to the added complexity, but then I'm just doing this for fun and it was an interesting challenge.

[Koston]

Pushed a new version to a separate development branch, which adds a new default "Simple Mode" that is fully usable with a joystick.

Code: Select all

curl -L https://github.com/Koston-0xDEADBEEF/MiSTer-ROMweasel/blob/current/romweasel.sh -o /media/fat/Scripts/romweasel.sh

[Koston]

Merged a bunch of changes back to main branch and marked v0.9.5 for public testing/use.

Code is a bit more mature now, I'm reaching limit on how much more time I have for playing around with this. And I've certainly long ago passed all thresholds of what's even remotely sensible to do in shellscript.


Note: I've added Sega Saturn CHD repository to it, but the core only supports loading BIN/CUE files for now.

[LamerDeluxe]

I've been testing the chd code and simple mode a bit and so far it has been working fine. This is a really handy script, thanks for all the effort you've put into it.

[Koston]

I've been testing the chd code and simple mode a bit and so far it has been working fine. This is a really handy script, thanks for all the effort you've put into it.

I'm happy to hear you're finding it useful.

I tested the chd sorting code against full PSX and MegaCD libraries, and there were only a handful that had so complicated or irregular file names that the logic can't handle. Exact figures are in git commit log.

(If) Going forward, I should really switch to another language and investigate using frameworks by other people. The game repositories have SQLite files, parsing those with access to more advanced data structures would be the smart thing to build on.

[LamerDeluxe]

I've been testing the chd code and simple mode a bit and so far it has been working fine. This is a really handy script, thanks for all the effort you've put into it.

I'm happy to hear you're finding it useful.

I tested the chd sorting code against full PSX and MegaCD libraries, and there were only a handful that had so complicated or irregular file names that the logic can't handle. Exact figures are in git commit log.

Sounds like a good enough solution.

(If) Going forward, I should really switch to another language and investigate using frameworks by other people. The game repositories have SQLite files, parsing those with access to more advanced data structures would be the smart thing to build on.

Would be great if it could be extended to include support for more cores in the future.

If the controller option could be expanded with a buttons for select and page up/down, that would also be really handy.

[Koston]

Would be great if it could be extended to include support for more cores in the future.

Most of the simple ones are already added. There might be a few more simple ones, but after that it means adding custom logic for each.

If the controller option could be expanded with a buttons for select and page up/down, that would also be really handy.

I investigated this a bit, before ending up implementing "Simple Mode". There are two ways how it could be done, if I'm not mistaken. The script could parse USB device node tree, fork event readers for each found controller event node and parse the event data. Safe to say, that's not happening.

Second way would be patching the main MiSTer binary to map unused buttons to keyboard events. No idea what chances such a patch would have to get merged upstream, so I'm a bit wary about spending a lot of time on it.