Individually bootable VHDs for each game

[Tchucolate]

I did find eXoDOS via a tutorial video, THAT is certainly the collection to start with, 7000 games, and close to a 500GB download that just completed
Now I need to whittle it down to just the games that "mattered", at least for me anyway

[Caldor]

But making individual VHDs for each game, it seems the VHD might as well include the system, at least if its DOS which I am thinking is the main intent anyway. DOS does not really take up more than a disk if its just to boot a game. A DOS 6.22 or 7.1 bootdisk can be made the same way I made the FreeDOS bootdisk. Especially when also adding sysctl and such to make the configurations even more specific.

But since there are thousands of DOS games, I still think it would make more sense to have several games on each VHD, making it into different collections, and maybe let each VHD have a DOS menu for launching these games. A good option for this is Launchbox for DOS I think.
https://forums.launchbox-app.com/files/ ... l-edition/

The better way might be to use a launcher which decide which individually memory setting are needed for the game (3 possibilities as mentioned before) and rename the relevant confignoems.sys to the global config.sys with autostart in autoexec and reboot the pc as an example. After quitting the game the config.sys has to be restored.

Problem is the launcher comes after having setup the memory. You cannot chose XMS or EMS after you booted and launched the launcher. To make this work, you would have to pick the game in a boot menu and have that set the memory settings and possibly directly into the game.

I mean, I guess the launcher could rename files as you mentioned... hmm... yes I guess that could work. Just always have 3 sets of configuration files ( or however many might be needed for the games on that VHD), and then just copy them and make them into the active autoexec and config.sys file. But should it then launch the game after rebooting? Swapping the config files around, would only work if its possible to reboot from a script and also have it launch the game. I guess... well, I think it would be overcomplicating things, but it could be done so that instead of launching a game it would copy in the boot files requires, and then edit the autoexec.bat file and add some lines specific for the game being launched, and also add lines to copy back the regular config files that launches the launcher on boot when quitting the game. If you just directly reset, it would then boot into that same game though, which would be a bit confusing I think.

[Caldor]

You mean something like this?
https://www.youtube.com/watch?v=tmSFSXE ... e=youtu.be

Wow! Sounds like a perfect solution for all games!

Oh yeah... that does seem like it should work for everything.

[Caldor]

Im wondering, can anyone recommend some sites, where these old DOS games can be downloaded. Ideally, these would be as raw folders and not wrapped in some kind of emulator. Though if the emulator were in a sibling folder that would not be too much of an issue.

I'd like to start making compilation VHDs, but I want to get on with getting the games in there and working, rather than searching the net for good sources of working games.

There are a lot of abandonware sites and there is the exodos collection. There are also a lot of the classic games on GOG, and they might be wrapped in an emulator. Sometimes it is a problem if its using ScummVM, but the ones that use DOSBox should not be a problem, you can just take the game files and use those.

[Captain FPGA]

Please if you do this create a special set of settings for Duke Nukem to keep it from crashing. Apparently others have got it to work without crashes, I assume they must've tweaked something.

I tried a litany of settings in both TDS and the Mister menu but nothing works. I just want to play this game with peace of mind. Don't worry about space I've got a 1TB micro SD card with 600+ GB free (been filling her up)

[Bas]

I've been doing this manually for a while. Now constructing a build pipeline that takes a config file, OS binaries and game binaries to cobble together a VHD. Works nicely and is a very kids-friendly way of packaging DOS games, but I hardly have time anymore these days and between my package mgt project and this.. progress is incredibly slow.

[Captain FPGA]

I've been doing this manually for a while. Now constructing a build pipeline that takes a config file, OS binaries and game binaries to cobble together a VHD. Works nicely and is a very kids-friendly way of packaging DOS games, but I hardly have time anymore these days and between my package mgt project and this.. progress is incredibly slow.

Think I speak for a lot of folks when I say we'll understand if you can't finish. We all have busy lives after all, it was thoughtful of you to even consider this at all.

I do hope it doesn't get too busy and you'll get to provide as many single VHD titles as your heart desires. Again, keeping my fingers crossed for a working Duke Nukem.

[Caldor]

I was thinking of solutions of doing this, possibly with some app to create VHD files in Windows using some C# app that creatures the configuration files and such for each game. Probably splitting up an existing one.

But... it just seems like a problematic solution. If you have 50+ games like this, it can become pretty annoying setting up. It becomes a lot of VHDs to browse. I already have some 20 VHDs, and already it seems like a lot to browse through, and the UI for the AO486 core is not really designed for it.

Maybe instead it could be different collections of games. Maybe based on developers or genres. With a boot menu like the 300 best games collection and such.

[Bas]

That's what I am building (slowly). Generating an actual MS-DOS MBR and partition table in a way that's completely independent of the host OS is pretty complicated. Turns out that MS-DOS had many variants of on-disk structures and I'm aiming for 100% accuracy here. I only have about 3 hours a week for this though, so it's slow work. My first target is 6.22. Anyone here have any authoritative sources for the actual on-disk structures for MS-DOS 3.1 and higher please DM me!

I'm looking to categorize and symlink according to a few different cross-sections, borrowing metadata from the eXoDOS collection.

The actual build pipeline currently runs on my home lab Jenkins box on FreeBSD. As soon as I have reasonable code working I'll push to GitHub under BSD license. That'll be some time though.

[gus0650]

Individual bootable VHD files for each game is what I have been waiting for in a long time. Awesome. Have you made any progress? When can i play some?

[Caldor]

With the shared folder now having been improved, I think it might make more sense to use boot disks instead, where each boot disk the boots specific games. But overall a problem with this solution, is that if you want to generally make changes to a boot, maybe there is a better driver for CD ROM drives, MIDI, sound card, graphics and so on, then it would have to be changed on each individual VHD. Another problem is needing to use the interface to select the specific game to boot out of all the VHDs you have.

There is the TOP 300 collection, which is just some of the DOS games... but there are thousands of DOS games, so it could quickly become rather overwhelming. But one of the tools released could fix that, because it should be possible with a script inside a BAT file, to switch VHDs and such, and control it all from one big boot menu.

But the advantage of boot disks and using the shared drive is the space needed. Because each game would only need the size of the games files. VHDs usually end up being larger than they need to be for each specific game.

I did recently try DOSBOX Pure, and it has an interesting feature, supporting ZIP files and booting from a ZIP file. I tried this using ZIP files from the ExoDOS collection. Works quite well, I think its just the game files zipped and then each zip file containing a BAT file with a specific name, which handles how the game should be run.

[Bas]

Progress is slow between job changes and other crap in my private life. Today I picked the old code up again and I'm now at the point where my tool writes a VHD structure, calculates CHS geometry for it and writes a valid MS-DOS MBR. Next step: partitioning, which is tiny. From there I intend to work with existing Linux tools to write the actual OS and the games to their respective disk images. This may sound very close, and I'm going to open source the VHD generator as soon as I work out the kinks and clean it up. Unfortunately it does feel like I'm still quite far from my goal due to severe lack of time.

[Caldor]

So the tool can write VHDs with various sizes? The next step being to be able to put data in the VHD? That would be very useful. I have tools that can mount a VHD, but those are GUI tools. Something that could do it with batch jobs and maybe help add files with a program would be very useful.

[Bas]

I'm writing a CLI tool that generates a valid VHD of a certain size in MB. Nothing qemu-img couldn't already do, but the main problem is to get the on-disk structures there for a bootable DOS environment. So I'm taking it a little bit further and writing an exact MS-DOS MBR (version 6.22 for now), a valid partition table, bootable primary partition (FAT16 for now) plus the bits in IO.SYS that need to be in a special place for DOS to boot at all.

After this, Linux can mount the partition and use its own tools to drop files in there. I'm now working on the partitioning, so making reasonable progress for a change.

[Caldor]

That would help making different VHDs for each game

[Bas]

Oh boy was I mistaken! Just about every modern tool/lib I'm evaluating is doing something different and taking liberties from the original disk management tools that came with our beloved retro OS'es. In modern times this is with good reason, but that's not our use case. I'm aiming for absolute accuracy here, which forces me to write my own code not just for the virtual disk containers and the MBR but also the *many* subtle variations of the FAT12/16 filesystem header that used to float around. Right now I'm boxing myself into just replicating MS-DOS 6.22 specifically but even that is more work than expected. So is there progress on my end? Yes, and it's going steadily, but for now I'm still trying to find the bottom of this rabbit hole in a few scarce hours each week and all while also teaching myself the Rust programming language in the process.

[Caldor]

I was thinking a bit about this and came to think about a feature in DOSBox Pure. It allows booting games from zip files. This feature would require a change to the AO486 core itself if it was to support it. I am thinking the way it would work would be that there would have to be a zip or VHD with a version of DOS for this feature to use. It should then make temp files for each game and all changes made would be saved to some temp files.

A problem with this approach is that the VHD or whatever the zip is booted from would probably not really support every possible game. DOSBox has some settings that just pretty much works for all games. Hmmm... probably too complicated to implement I guess. Just seems nice if it would be possible to run games from zip files in that way. It would also make DOS 6.22 more than enough as you would not really need bigger than 2gb partitions if you run each game on its own.

[Bas]

Small progress report. I'm up to the point where my program generates a disk images sized between 5 and 500 MB with a valid partition table and a single active partition that's recognized by MS-DOS. When I use DOS to format this and copy over a game, it works. I'm now at probably 30% of the way to replicating what FORMAT /S does, which is absolutely key to the whole idea. After that it's mostly downhill and existing tools can fill the gaps.

I'm between jobs right now so I have time and am making good progress now. I'll be open sourcing my work when my new job starts but I hope to get something real going and show it before that. Job starts October 3rd, so I'll report back around then.

[thorr]

This is slightly off topic but still relevant as it pertains to the ao486 portion. I am getting dangerously close to completing my arcade cabinet that I have been working on periodically for a couple of years now and this thread got my attention. There are a few arcade-like PC games that I want to have in an easy list I can pick from a menu. Is there a way to customize the MiSTer menu to just have a list of all the games you want to play from any system on one menu list? I would have all the arcade games, some games from consoles, and some games from ao486. I would love to be able to do this with one list, maybe with subfolders to sort things. Maybe all this can live under scripts or something, and maybe there is a way to have it auto launch the scripts folder when bringing up the menu. There is probably a discussion about this somewhere already. If anyone can point me to that I would appreciate it.

Edit: Actually after thinking about it, I could probably write a single script myself to do this if I knew the command line syntax to launch titles with specified ROM's, or VHD's in ao486's case. Or maybe one of those existing fancy front ends would work like Launchbox since Linux is on the MiSTer.

Edit 2: I think this is the thread I am looking for: viewtopic.php?t=2493

[akeley]

Edit 2: I think this is the thread I am looking for: viewtopic.php?t=2493

Try this one: viewtopic.php?t=4470

[Caldor]

Now that there is the PCXT core, this might become relevant for that core as well It just got better floppy support but still lacks mouse support. Progress has been made towards that as well though.

Could you share your tools? I might be able to help at some point creating VHDs. I do have a large collection of DOS games and am familiar with scripting and coding.

[Bas]

I will share my work-in-progress by the end of this week. Initially I had wanted to open-source the toolbox when it reaches a minimally viable state, so that it'd be at least structurally sane and usable for a minimal purpose. I probably won't achieve that goal before next Sunday. I'm now focusing on a few things:

1. Documenting things so that they're understandable to all who can work with Rust.
2. Finish the implementation of FAT16 either by using an existing crate, or by coding the bare minimum myself.

The main issue here is in point 2. Like I said, the tool now creates a disk image of any size up to the old ~528MB LBA limit and deploys a single primary partition on there that fills the "disk". When using MS-DOS 6.22, these images work fine. I can format the partition, Scandisk doesn't complain about anything anymore, and after a FORMAT /S, the system boots from them smoothly.

Existing Rust crates for implementing FAT, however, do not follow Microsoft very closely but base off other implementations like the Linux vfat driver or other people's reverse engineering efforts. Those efforts all had the focus of using FAT in production in some sort of form, on modern systems. What we're doing here, is reimplement FAT as it worked decades ago as if we were flashed back to that time period and none of those modern systems exist yet.

TLDR: I need a way to do what MS-DOS 6.22's FORMAT /S does, but independently from MS-DOS, so that I can integrate that into a CI/CD build pipeline. The sad bit: Microsoft doesn't document any of that in a sufficiently detailed way and FreeDOS has drifted too far away from it, and I'm still an absolute n00b at programming things that are this low-level.

The steps after that:

3. Formulate a sort of recipe/metadata format that can be read by a CI/CD pipeline and used to write the game + support files like mouse drivers, tweaked DOS extenders etc. to the image, and compose files like CONFIG.SYS and AUTOEXEC.BAT in an automated way. The way I see this now, Ansible looks like a decent candidate for doing this. A specific role could abstract away the nitty-gritty, and we'd end up with a simple-to-write YAML manifest per game. Now I've been living and breathing Ansible in my day job for years, so I really hope to get over the FAT bump soon.

4. Run all of these manifests through a pipeline script. The result, effectively, should be a 1-to-1 mapping of YAML manifest to a resulting VHD.

Why do it this way? Reduction of toil. Anyone and everyone can duplicate this, tweak the YAML, the Ansible role or the Rust codebase for the image generator, and auomatically generate new VHD's on the spot. This takes out masses of manual toil that must be repeated whenever someone finds an improvement, the core evolves in some incompatible way, or someone figures out how to run a particular game that didn't work before.

[AmintaMister]

I will share my work-in-progress by the end of this week. Initially I had wanted to open-source the toolbox when it reaches a minimally viable state, so that it'd be at least structurally sane and usable for a minimal purpose. I probably won't achieve that goal before next Sunday. I'm now focusing on a few things:

1. Documenting things so that they're understandable to all who can work with Rust.
2. Finish the implementation of FAT16 either by using an existing crate, or by coding the bare minimum myself.

The main issue here is in point 2. Like I said, the tool now creates a disk image of any size up to the old ~528MB LBA limit and deploys a single primary partition on there that fills the "disk". When using MS-DOS 6.22, these images work fine. I can format the partition, Scandisk doesn't complain about anything anymore, and after a FORMAT /S, the system boots from them smoothly.

Existing Rust crates for implementing FAT, however, do not follow Microsoft very closely but base off other implementations like the Linux vfat driver or other people's reverse engineering efforts. Those efforts all had the focus of using FAT in production in some sort of form, on modern systems. What we're doing here, is reimplement FAT as it worked decades ago as if we were flashed back to that time period and none of those modern systems exist yet.

TLDR: I need a way to do what MS-DOS 6.22's FORMAT /S does, but independently from MS-DOS, so that I can integrate that into a CI/CD build pipeline. The sad bit: Microsoft doesn't document any of that in a sufficiently detailed way and FreeDOS has drifted too far away from it, and I'm still an absolute n00b at programming things that are this low-level.

The steps after that:

3. Formulate a sort of recipe/metadata format that can be read by a CI/CD pipeline and used to write the game + support files like mouse drivers, tweaked DOS extenders etc. to the image, and compose files like CONFIG.SYS and AUTOEXEC.BAT in an automated way. The way I see this now, Ansible looks like a decent candidate for doing this. A specific role could abstract away the nitty-gritty, and we'd end up with a simple-to-write YAML manifest per game. Now I've been living and breathing Ansible in my day job for years, so I really hope to get over the FAT bump soon.

4. Run all of these manifests through a pipeline script. The result, effectively, should be a 1-to-1 mapping of YAML manifest to a resulting VHD.

Why do it this way? Reduction of toil. Anyone and everyone can duplicate this, tweak the YAML, the Ansible role or the Rust codebase for the image generator, and auomatically generate new VHD's on the spot. This takes out masses of manual toil that must be repeated whenever someone finds an improvement, the core evolves in some incompatible way, or someone figures out how to run a particular game that didn't work before.

Congratulations, you're doing a great and invaluable job!

[Bas]

Small success: I just managed to automagically format my first generated VHD image. It uses the fatfs crate so it's not absolutely MS-DOS 6.22 accurate, but it'll do for the MVP stage. Having working games and a working build pipeline trumps purism over accuracy for now, so I'm proceeding to making the images bootable now.