File Size For Entire Library of Cores and ROMs on MiSTer?

Just got a MiSTer and need some help? Join the forum and post your questions here!
User avatar
Captain FPGA
Posts: 371
Joined: Sun Apr 11, 2021 9:19 pm
Has thanked: 195 times
Been thanked: 25 times

File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by Captain FPGA »

Howdy!

What does the entire library of cores and roms amount to for the MiSTer FPGA? Would it come to a TB or two?

I'm shopping around for micro sd cards, but I'm not sure how large of a storage I'll need.
User avatar
aberu
Core Developer
Posts: 1144
Joined: Tue Jun 09, 2020 8:34 pm
Location: Longmont, CO
Has thanked: 244 times
Been thanked: 388 times
Contact:

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by aberu »

Captain FPGA wrote: Sun Apr 11, 2021 9:36 pm Howdy!

What does the entire library of cores and roms amount on the MiSTer FPGA? Would it come to a TB or 2?

I'm shopping around for micro sd cards but I'm not sure how large of a storage I'll need.
For everything that every single core can run? This includes ao486, so you are probably looking at many many TB, since the library of ao486 games is huge. Amiga games too.

If you think your use-case will be having everything ever made then you should get an external powered USB HDD that is like 8TB at least. Especially if you are hoping to have the PS1 library in full (which is currently taking up almost 3TB on my storage). Add in the Saturn library as that possibility improves day by day... you will exceed even an 8TB drive. the ao486 top300 is over 100GB, the top500 will be more.

So I think that your goal to have every single thing ever for every single system is an interesting one, to say the least. :D

Before PS1/Saturn, you are probably looking at 2TB max or so, just with "best of" Amiga and "best of" ao486 Top 300 images that are out there, not even their full library.
birdybro~
zakk4223
Posts: 270
Joined: Sun May 24, 2020 10:55 pm
Been thanked: 107 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by zakk4223 »

eXoDOS alone is nearly 600GB.

I have a fairly complete set of everything and it fits on a 512GB, but that's without any DOS stuff. Also the Amiga is MegaAGS (so probably not 'complete') and the Atari ST is likely a 'curated' image. I have a feeling if you went down the deep rabbit hole for all the computer systems the size would grow dramatically
User avatar
Captain FPGA
Posts: 371
Joined: Sun Apr 11, 2021 9:19 pm
Has thanked: 195 times
Been thanked: 25 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by Captain FPGA »

Whoa! Here I'm thinking these old ass machines were like MB that needed to be converted into certain images and zipped to reduce file size. I had no idea I was dealing with the raw libraries for these items. As for the PC systems libraries, I under-estimated.

Narrowing my search for now, how many GB are we looking at with the golden age consoles/arcade? (with some PC stuff sprinkled in) I'm talking Mega Drive/Genesis, Mega CD, 32X, SNES, NES, Neo Geo, SMS, Neo Geo CD, CPS1, CPS2, Windows XP, '95, '98, 3.1x, MSX, SS16, Gameboy, Game Gear, GBA.

I know there's a real possibility Saturn, PS1, 3DO and Dreamcast (I know long shot...) could come later. But I don't see that happening anytime soon. Right now I can afford a 512GB or MAYBE a 1TB SD card. If 512 holds all of the above, all the better.
User avatar
bazza_12
Top Contributor
Posts: 404
Joined: Sun May 24, 2020 7:49 pm
Location: Yorkshire, UK
Has thanked: 247 times
Been thanked: 112 times
Contact:

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by bazza_12 »

at the moment you can get pretty much everything (not exodos) including Flynns Top 300 on a 256gig card, (if the roms are zipped and the cd's are CHD'd). I also have a 128gig card on a usbstick for additional amiga builds and Flynns a0486 1980-2006 collection, plus various windows builds. So if you can.. get a 512gig card.. (i also have a 1TB usb drive as a backup)
The music is reversible but time is not. Turn back. Turn back
jvas
Posts: 47
Joined: Tue Aug 25, 2020 8:15 am
Has thanked: 5 times
Been thanked: 2 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by jvas »

Don't forget: Playstation is coming to town....
grizzly
Posts: 375
Joined: Tue Jun 16, 2020 12:22 pm
Has thanked: 55 times
Been thanked: 76 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by grizzly »

Captain FPGA wrote: Mon Apr 12, 2021 12:39 am Narrowing my search for now, how many GB are we looking at with the golden age consoles/arcade? (with some PC stuff sprinkled in) I'm talking Mega Drive/Genesis, Mega CD, 32X, SNES, NES, Neo Geo, SMS, Neo Geo CD, CPS1, CPS2, Windows XP, '95, '98, 3.1x, MSX, SS16, Gameboy, Game Gear, GBA.
That was not narrowing down at all.
Adding Windows XP, '95, '98, 3.1x to the mix will mean many,many,MANY more TB of data since NONE of those where involved in the earlier answers.
As for me i do have mostly full sets for everything except for Amiga/DOS (i do have MegaAGS on SD card and eXoFlynn Top 300 DOS Pack on NAS) and is around 400GB.
The eXoFlynn Top 300 DOS Pack does NOT fit on my 512GB SDcard (with everything else) so is on my NAS.
User avatar
bazza_12
Top Contributor
Posts: 404
Joined: Sun May 24, 2020 7:49 pm
Location: Yorkshire, UK
Has thanked: 247 times
Been thanked: 112 times
Contact:

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by bazza_12 »

Captain FPGA wrote: Mon Apr 12, 2021 12:39 amNarrowing my search for now, how many GB are we looking at with the golden age consoles/arcade? (with some PC stuff sprinkled in) I'm talking Mega Drive/Genesis, Mega CD, 32X, SNES, NES, Neo Geo, SMS, Neo Geo CD, CPS1, CPS2, Windows XP, '95, '98, 3.1x, MSX, SS16, Gameboy, Game Gear, GBA.
mine are as follows:-

Mega Drive/Genesis (inc 32x) - 6gig
Mega CD - 37gig
SNES - 6gig
NES - 1gig
NeoGeo - 7gig
SMS - 350meg
MSX - 3gig
Gameboy - 3gig
GameGear - 322meg
GBA - 13gig

WIN 95, 98, 3.1vhds - 2gig each
The music is reversible but time is not. Turn back. Turn back
User avatar
Captain FPGA
Posts: 371
Joined: Sun Apr 11, 2021 9:19 pm
Has thanked: 195 times
Been thanked: 25 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by Captain FPGA »

bazza_12 wrote: Mon Apr 12, 2021 8:27 pm
Captain FPGA wrote: Mon Apr 12, 2021 12:39 amNarrowing my search for now, how many GB are we looking at with the golden age consoles/arcade? (with some PC stuff sprinkled in) I'm talking Mega Drive/Genesis, Mega CD, 32X, SNES, NES, Neo Geo, SMS, Neo Geo CD, CPS1, CPS2, Windows XP, '95, '98, 3.1x, MSX, SS16, Gameboy, Game Gear, GBA.
mine are as follows:-

Mega Drive/Genesis (inc 32x) - 6gig
Mega CD - 37gig
SNES - 6gig
NES - 1gig
NeoGeo - 7gig
SMS - 350meg
MSX - 3gig
Gameboy - 3gig
GameGear - 322meg
GBA - 13gig

WIN 95, 98, 3.1vhds - 2gig each
Perfect thanks! That helps me out a lot!
grizzly wrote: Mon Apr 12, 2021 5:44 pm
Captain FPGA wrote: Mon Apr 12, 2021 12:39 am Narrowing my search for now, how many GB are we looking at with the golden age consoles/arcade? (with some PC stuff sprinkled in) I'm talking Mega Drive/Genesis, Mega CD, 32X, SNES, NES, Neo Geo, SMS, Neo Geo CD, CPS1, CPS2, Windows XP, '95, '98, 3.1x, MSX, SS16, Gameboy, Game Gear, GBA.
That was not narrowing down at all.
Adding Windows XP, '95, '98, 3.1x to the mix will mean many,many,MANY more TB of data since NONE of those where involved in the earlier answers.
As for me i do have mostly full sets for everything except for Amiga/DOS (i do have MegaAGS on SD card and eXoFlynn Top 300 DOS Pack on NAS) and is around 400GB.
The eXoFlynn Top 300 DOS Pack does NOT fit on my 512GB SDcard (with everything else) so is on my NAS.
Heh, then let's call it clarifying. It looks like I'll be able to fit just about everything I want though.

I'll be going for one of these micro sd cards: https://www.amazon.com/dp/B07P9W5HJV/
Dreams don't die!
Image
User avatar
Chris23235
Top Contributor
Posts: 841
Joined: Sun May 24, 2020 8:45 pm
Has thanked: 106 times
Been thanked: 167 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by Chris23235 »

Don't forget the 200GB PCE-CD Rom images they are definitely worth it.
User avatar
Captain FPGA
Posts: 371
Joined: Sun Apr 11, 2021 9:19 pm
Has thanked: 195 times
Been thanked: 25 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by Captain FPGA »

Chris23235 wrote: Wed Apr 14, 2021 7:49 am Don't forget the 200GB PCE-CD Rom images they are definitely worth it.
WTF the PC Engine Duo CD library is that huge in GB?! Here I thought they were Sega CD or Neo Geo CD size.
Dreams don't die!
Image
elvis
Posts: 62
Joined: Sun May 24, 2020 9:25 pm
Has thanked: 41 times
Been thanked: 35 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by elvis »

How long is a piece of string?

From my NAS, retro games only, which the MiSTer mounts on boot:

Code: Select all

root@server:~# du -shx /data/games
2.5T    /data/games
However thanks to BtrFS (a much better filesystem than exFAT), inline zstd compression and deduplication shrinks the actual used data far better than zip. About 1.6TB transparently, no decompression needed by MiSTer (it thinks it's seeing the full 2.5TB uncompressed).

Code: Select all

root@server:~# du -shx /data/games
2.5T    /data/games
root@server:~# compsize -x /data/games
Processed 417955 files, 9164013 regular extents (14660514 refs), 17612 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       80%      1.6T         2.0T         2.5T
none       100%      1.2T         1.2T         1.4T
zstd        46%      365G         781G         1.1T
User avatar
bazza_12
Top Contributor
Posts: 404
Joined: Sun May 24, 2020 7:49 pm
Location: Yorkshire, UK
Has thanked: 247 times
Been thanked: 112 times
Contact:

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by bazza_12 »

Chris23235 wrote: Wed Apr 14, 2021 7:49 amDon't forget the 200GB PCE-CD Rom images they are definitely worth it.
PCE-CD rom, as in Turbografx16-cd?
The music is reversible but time is not. Turn back. Turn back
User avatar
aberu
Core Developer
Posts: 1144
Joined: Tue Jun 09, 2020 8:34 pm
Location: Longmont, CO
Has thanked: 244 times
Been thanked: 388 times
Contact:

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by aberu »

Here's a rough breakdown of the big boys if you wanted "everything" and in redump matching fashion. Looking at my beefy boye hard drive for this real quick...

1. Every single PS1 game = well over 3TB unzipped in cue/bin
2. Every single Mega CD game = 186GB unzipped in cue/bin
3. TurboGrafx16 CD/PC-Engine CD = 184GB unzipped in cue/bin
4. eXoDOS = over 600GB unzipped, would have to span 150 separate virtual hard drives in the /media/$device/games/ao486 folder
5. Hardware Target Game Database (https://github.com/frederic-mahe/Hardwa ... e-Database) MiSTer FPGA approved packs (which have quite a few repeats of files for organizing purposes - 209GB

There's obviously going to be more, like I said before, Amiga and other PC cores are nowhere near complete in this list.
bazza_12 wrote: Wed Apr 14, 2021 11:53 am
Chris23235 wrote: Wed Apr 14, 2021 7:49 amDon't forget the 200GB PCE-CD Rom images they are definitely worth it.
PCE-CD rom, as in Turbografx16-cd?
Yup, they are the same thing, just different names in the west than in Japan.

CUE/BIN full redump unzipped is 184GB btw, not 200GB.
birdybro~
User avatar
bazza_12
Top Contributor
Posts: 404
Joined: Sun May 24, 2020 7:49 pm
Location: Yorkshire, UK
Has thanked: 247 times
Been thanked: 112 times
Contact:

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by bazza_12 »

This is probably a silly question but is the primary SDCard slot faster reading/writing that a USB SSD?
The music is reversible but time is not. Turn back. Turn back
commander
Posts: 27
Joined: Mon May 25, 2020 6:56 am
Has thanked: 2 times
Been thanked: 2 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by commander »

you can save a amount of space when converting sega-cd,tg16-cd to chd format
have fun
User avatar
Chris23235
Top Contributor
Posts: 841
Joined: Sun May 24, 2020 8:45 pm
Has thanked: 106 times
Been thanked: 167 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by Chris23235 »

bazza_12 wrote: Wed Apr 14, 2021 3:46 pm This is probably a silly question but is the primary SDCard slot faster reading/writing that a USB SSD?
A USB SSD is faster in theory but it shouldn't matter much, because a) the speed is limited by the USB 2.0 port and b) most systems that load a lot (CD-Systems) load with their original speed.
elvis
Posts: 62
Joined: Sun May 24, 2020 9:25 pm
Has thanked: 41 times
Been thanked: 35 times

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by elvis »

bazza_12 wrote: Wed Apr 14, 2021 3:46 pm This is probably a silly question but is the primary SDCard slot faster reading/writing that a USB SSD?
The media speed itself will likely have more impact on that. MicroSD cards are sold in speed classes such as V10 (10MB/s), V30 (30MB/s), and so on.

I don't actually know the specs of the DE10Nano's MicroSD slot, so I'll try to look that up and get a real world number for you.

The DE10Nano's USB system is USB2, with a maximum theoretical bounds of 480mbit/s (60MB/s max theoretical), but worth remembering that USB2 storage is half duplex, so the "feel" of it is far worse for use as a general purpose file system (where small writes happen even on read operations, to update the atime [access time]).

By far the fastest storage IO to the device would be the Gigabit Ethernet, at 1000mbit/s, or 125MB/s max theoretical.
User avatar
bazza_12
Top Contributor
Posts: 404
Joined: Sun May 24, 2020 7:49 pm
Location: Yorkshire, UK
Has thanked: 247 times
Been thanked: 112 times
Contact:

Re: File Size For Entire Library of Cores and ROMs on MiSTer?

Unread post by bazza_12 »

fascinating, I was trying to work out if there was an difference in access time in having massive files, like iso/cd etc on USB SSD/HD instead of directly on an SDCard. Personally I found having VHD files on USB HD (not SSD) caused a reduction in access time.. I wouldn't mind getting a USB SSD to compare.. seems like so much more than the USB speed are determining factors.. thanks for the answers :)
The music is reversible but time is not. Turn back. Turn back
Post Reply