DOSContainer Alpha Release

Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

I'm taking a small detour now. The current state of DOSContainer allows me to package just about anything from the dawn of time until about 1985. That's where I'll start, and I want the process be friendly to outside contribution. So in short:

  1. I'm completing the zipfile downloader feature on DOSContainer to enable the layered build architecture that I envisioned from the start.
  2. Set up robust and fast hosting for everything. (done)
  3. Create a friendly webform for contributed games and YAML's.
  4. Publish tested and validated games on a new DOSContainer website.
  5. Put DOSContainer itself behind a web form. Upload your YAML, it spits out a VHD just for you. (stretch goal)

Once that flow is up, I'm starting in 1981 and working my way up through history. Implementing DOS versions and FAT variations follows as I reach the games for which they are period appropriate. Starting the packaging early helps me learn and figure out optimal workflow.

Another feature that I feel is a hard requirement when games become configurable themselves (1985-ish), is a "pull" option. This means you run a VHD on MiSTer and have a shell open at the same time. As you make the needed changes in the DOS environment to set up all the different hardware settings, you run a simple "pull" command on the VHD every time to extract a delta that captures only the config changes you made in DOS (yes, I think I can detect open/uncommitted files with reasonable confidence). Those changes then become a layer in the YAML. Once done, the tool wraps the whole lot up in a zipfile and a YAML manifest. The goal: instantly allow custom builds that mix and match all settings for the game itself where it makes sense. Just tweak the YAML to suit your wishes, or just build VHD's for every permutation you can think of in the blink of an eye. This should fix the huge amount of manual toil to configure each individual game.

By the way: this is all a hobby and I only have a few hours each week for it. I do welcome any help I can get in this! If you know Rust, awesome! If you're good with AWS serverless code: please raise your hand. If you want to design a website, logo or anything else: yay! I am very partial to static hosting with things like Hugo due to the low maintenance and no security/maintenance headaches like WordPress would bring.

Another option that I'm very open to, is to join with the 0Mhz effort and provide the plumbing for that. My own scope is wider than MiSTer itself.

AmintaMister
Posts: 300
Joined: Thu Sep 16, 2021 10:54 pm
Has thanked: 784 times
Been thanked: 48 times

Re: DOSContainer Alpha Release

Unread post by AmintaMister »

Bas wrote: Sat Apr 13, 2024 8:24 am

I'm taking a small detour now. The current state of DOSContainer allows me to package just about anything from the dawn of time until about 1985. That's where I'll start, and I want the process be friendly to outside contribution. So in short:

  1. I'm completing the zipfile downloader feature on DOSContainer to enable the layered build architecture that I envisioned from the start.
  2. Set up robust and fast hosting for everything. (done)
  3. Create a friendly webform for contributed games and YAML's.
  4. Publish tested and validated games on a new DOSContainer website.
  5. Put DOSContainer itself behind a web form. Upload your YAML, it spits out a VHD just for you. (stretch goal)

Once that flow is up, I'm starting in 1981 and working my way up through history. Implementing DOS versions and FAT variations follows as I reach the games for which they are period appropriate. Starting the packaging early helps me learn and figure out optimal workflow.

Another feature that I feel is a hard requirement when games become configurable themselves (1985-ish), is a "pull" option. This means you run a VHD on MiSTer and have a shell open at the same time. As you make the needed changes in the DOS environment to set up all the different hardware settings, you run a simple "pull" command on the VHD every time to extract a delta that captures only the config changes you made in DOS (yes, I think I can detect open/uncommitted files with reasonable confidence). Those changes then become a layer in the YAML. Once done, the tool wraps the whole lot up in a zipfile and a YAML manifest. The goal: instantly allow custom builds that mix and match all settings for the game itself where it makes sense. Just tweak the YAML to suit your wishes, or just build VHD's for every permutation you can think of in the blink of an eye. This should fix the huge amount of manual toil to configure each individual game.

By the way: this is all a hobby and I only have a few hours each week for it. I do welcome any help I can get in this! If you know Rust, awesome! If you're good with AWS serverless code: please raise your hand. If you want to design a website, logo or anything else: yay! I am very partial to static hosting with things like Hugo due to the low maintenance and no security/maintenance headaches like WordPress would bring.

Another option that I'm very open to, is to join with the 0Mhz effort and provide the plumbing for that. My own scope is wider than MiSTer itself.

This is an extraordinary effort, a perfect strategy, and a fundamental openess to collaboration, THANKS!

Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

Build 64 actually builds a working (in PCem, MiSTer untested) bootable Alley Cat VHD based on IBM PC DOS 2.00.

This build is a partly hardcoded proof of concept that should prove a number of items. I would VERY MUCH appreciate some testing of the below functionality:

  • Does this even run properly on Windows at all? I don't know, I don't have Windows.
  • Does the ARMv7 build run properly on the MiSTer? (stretch goal: 32-bit Raspberry Pi's should work in theory.. test please?)
  • This is the first build with the ZIP downloader/extraction code hooked into the build phase.
  • The downloader pulls files in over HTTPS from my under heavy construction AWS-based hosting location.

So please, anyone adventurous, have a look at build 64 and see how it goes. Use the YAML I posted above as an input file. The result should be alleycat.vhd which should work as advertised.

WARNING: This build contains a command to delete a single file. What it does is it downloads the zip file for Alley Cat, extracts that to a platform-appropriate temp location and then deletes the zip file. While this should all be very safe (I'm using standard-fare Rust crates for the handling of tempfiles/dirs, my own tests on Linux turned out fine) I can not guarantee anything here. Do NOT test this build on the master control station of your in-house nuclear reactor please, or on any other system you are particularly attached to. Chances of anything going badly wrong are practically non-existent, but I do feel I should warn you about being a guinea pig.

To satisfy some people's curiosity. The binaries for IBM DOS are currently still compiled as raw bytes into the executable itself. It was done this way because I didn't have a working downloader yet. When this build proves stable, my next step will be to remove these embedded files and make them into downloads just like the game files. When that's done you'll start seeing multiple progress bars as a VHD-build proceeds.

I do like how going from zero to working bootable VHD including this tiny game's download is done in 0.16 seconds on my laptop. This opens the possiblity of stuffing DOSContainer into an AWS Lambda function eventually.

Attachments
doscontainer-64.zip
DOSContainer build 64.
(6.18 MiB) Downloaded 26 times
rcblanke
Posts: 7
Joined: Thu Feb 09, 2023 7:45 pm
Been thanked: 8 times

Re: DOSContainer Alpha Release

Unread post by rcblanke »

Hi Bas,
Appears to be working fine on Win11:

win11 output
win11 output
doscontainer.png (108.95 KiB) Viewed 5400 times

though as you can see leaving out the .yml extension results in a funky message. Please note that I haven't tried to actually run the vhd on PCem or MiSTer.

output alleycat.vhd:
MD5 9cb6b2bb9d315726a4b6ab7de5a401a1
SHA 0fffca953b3243063b4059c0cd09a35ad4ba173f

Regards,
Ronald

User avatar
NML32
Posts: 297
Joined: Sun May 24, 2020 6:57 pm
Has thanked: 287 times
Been thanked: 50 times

Re: DOSContainer Alpha Release

Unread post by NML32 »

Confirmed working on Windows 10
Used doscontainer.exe windows_x86_64

Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

Thanks for your testing! The error messages are funky because you're looking at Rust passing the error down the chain and giving up because I didn't yet write a nice formatter for it yet. That's my bad, but mostly cosmetic. You giving me checksums makes a huge amount of sense, but I just realized they aren't meaningful yet. Files are dropped into the VHD with the current system date/time stamped on them so every VHD will in fact be slightly different. I hadn't thought of that yet (just needed some kind of date/time to work and even that is wrong right now). I'll pick that up as an issue and use the source files' timestamps so builds become reproducible and actually correct.

User avatar
NML32
Posts: 297
Joined: Sun May 24, 2020 6:57 pm
Has thanked: 287 times
Been thanked: 50 times

Re: DOSContainer Alpha Release

Unread post by NML32 »

I get this error when running from MiSTer

root@MiSTer:/media/fat/doscontainer # ./doscontainer build allycat.yaml
./doscontainer: /lib/libc.so.6: version GLIBC_2.32' not found (required by ./doscontainer)
./doscontainer: /lib/libc.so.6: version
GLIBC_2.33' not found (required by ./doscontainer)
./doscontainer: /lib/libc.so.6: version `GLIBC_2.34' not found (required by ./doscontainer)

User avatar
NML32
Posts: 297
Joined: Sun May 24, 2020 6:57 pm
Has thanked: 287 times
Been thanked: 50 times

Re: DOSContainer Alpha Release

Unread post by NML32 »

Success with build #68 creating allycat.vhd from MiSTer FPGA.

https://youtu.be/4ghmIWLhk_M

Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

Wow, that was fast! I changed the build target over from glibc to musl, that helps apparently. Impressed with the speed of this thing on dinky MiSTer ARM!!

Last build for today will probably be 72 if it succeeds in about 15 minutes or so, after that it's back to work for a whole week again so not sure how much time I'll have to keep going. The biggest hurdle now is to support subdirectory's on the VHD. I did clean up and expand the documentation, added a section on the updated YAML format. I also do fully intend to support the Bash version's YAML format as well, but things may break there for a while. I will port over any games from the old version to the new though.

Change log lives here: https://code.area536.com/DOSContainers/ ... ranch/main

Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

Just a quick update on progress. Lots of regression as I'm generalizing the method of writing files/dirs to the file system. So while it seemed that build 64 was really close, I had to rip out ever growing parts of the file system code to allow for the general pattern of: extract this ZIP file to a temp location, then copy everything from that location over to the VHD recursively.

Why is this difficult? Mainly because ancient DOS versions have divergent ideas on how to write the FAT's root directory versus regular subdirectories. Flexibility in the core code is required but that runs the risk of creating a ball of spaghetti.

Writing a file on FAT entails allocating clusters for it, checking and creating the correct directory entry for it, and then writing the bytes to the previously allocated clusters in the data area of the partition/volume. What I'm seeing now is slight fluctuations on where the data area is supposed to actually begin.

Now I remember the rumors back in the day of how Commodore and Olivetti supposedly weren't exactly IBM compatible but I never got more than amecdotal hearsay on those. As this project progresses I get to see a lot of the icky underbelly of how MS-DOS was marketed before MS released its own builds directly. Great stuff for a blog series, but very annoying right now.

User avatar
limi
Top Contributor
Posts: 639
Joined: Sun May 24, 2020 6:53 pm
Has thanked: 147 times
Been thanked: 448 times

Re: DOSContainer Alpha Release

Unread post by limi »

Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

Interesting. This looks a lot more complete than the previous code, which was a steaming mess of versions all mixed together.

User avatar
RealLarry
Top Contributor
Posts: 781
Joined: Mon May 25, 2020 4:04 am
Location: San Junipero/DE/Earth
Has thanked: 96 times
Been thanked: 342 times

Re: DOSContainer Alpha Release

Unread post by RealLarry »

If anyone is interested: MS published their MS-DOS v4.0 on GitHub.

Contributor of tty2oled, author of tty2tft, tty2rpi and update_tty2xxx
Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

Build 92 just came down the pipes. It's functionally almost identical to build 64 from earlier. However:

  • The YAML format changed and is now far more sensible and more in line with what is in the enclosed manual.
  • The OS files now also get downloaded instead of being hardcoded into the binary.
  • The downloader fully works and is now wired into the application properly. Use your own URL's if you want, any ZIP over http(s) is fair game. No more depending on my infra to package your games*.
  • Lots of cleanups and restructuring that enables the next steps.

The path is now clear for me to do two absolutely crucial things:

  • I̶m̶p̶l̶e̶m̶e̶n̶t̶ s̶u̶p̶p̶o̶r̶t̶ f̶o̶r̶ s̶u̶b̶d̶i̶r̶e̶c̶t̶o̶r̶i̶e̶s̶. Done!
  • Implement support for FAT16 and FAT16B

So what can you do with this already? Well.. package really simple games like Alley Cat, that do not require subdirs to work. Everything now just gets plonked into C:\ and can run from there if you give it your own AUTOEXEC.BAT inside the ZIP. This is very limited and looks like a mess, but functionally should be just fine for the early 1980's when games didn't have many config options yet anyway.

A new test YAML for Alley Cat:

Code: Select all

---
metadata:
  title: Alley Cat
  publisher: IBM
  year: 1984
  comment: Testing DOSContainer, not functional yet.
  filename: alleycat.zip
diskname: alleycat.vhd
disktype: pcxt
disksize: 10
application:
  - url: https://dosk8s-dist.area536.com/alleycat.zip
    label: Alley Cat
os: IBMDOS200

And for some more proof of concept ancient history:
https://github.com/doscontainer/games/

Yes, I will eventually port all the games that were already packaged for the Bash version of DOSContainer.

*Yes, for OS downloads these builds still do depend on my infra. Just you go an try beat AWS S3+CloudFlare for serving out a tiny ZIP file though.. it'll be there, trust me. And for the future, yes, you get the freedom of overriding the OS zips as well.

Attachments
doscontainer-92.zip
(7 MiB) Downloaded 28 times
Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

I just cracked the puzzle on how to recursively create subdirs. Filling them with files shouldn't be rocket surgery, just a lot of grunt work. To be continued. Excited!

..learning a lot about memory allocation, memory safety and concurrency. I'm actually glad I picked Rust over other languages. It's one thing to write up a tree structure in a language like Python. It's a whole different matter to do so correctly while working with Rust's borrow checker breathing down your neck. Really appreciating the elegance of an Arena allocator even though it means scrapping and reworking half my FAT code. Things are working out for the better.

Bas
Top Contributor
Posts: 564
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 74 times
Been thanked: 271 times

Re: DOSContainer Alpha Release

Unread post by Bas »

Puzzle cracked I said last week.. right? Well, yes, theoretically. But just a minute ago: I created my first dir within a dir completely programmatically and scandisk does not complain anymore for any hdd size currently in scope. Now I can finally start writing functions that copy files over from the host to the VHD recursively into their respective subdirs. This, too, may still take a while due to a plethora of error conditions that need to be handled but there is light at the end of this tunnel and it's not a freight train coming at me! No build linked because this new functionality isn't user facing yet. Want to package ancient games from 1981-1984 without any subdirs? Use build 92 for now. Zork II was kind of fun!

Post Reply