How can I let MiSTer support Chinese Fonts?

[yaoese]

How can I let mister support Chinese Fonts?

[Newsdee]

If I'm not mistaken, the font only supports up to 128 or 256 characters, and each character can only be 8 pixels wide... so it would be difficult to have full support of Chinese characters.

[aberu]

That's correct.

https://github.com/MiSTer-devel/Fonts_MiSTer

The current framework only supports fonts like these.

Misc_Final_Fantasy.png (237.01 KiB) Viewed 5846 times

https://en.wikipedia.org/wiki/CJK_characters#Encoding

The number of characters required for complete coverage of all these languages' needs cannot fit in the 256-character code space of 8-bit character encodings, requiring at least a 16-bit fixed width encoding or multi-byte variable-length encodings. The 16-bit fixed width encodings, such as those from Unicode up to and including version 2.0, are now deprecated due to the requirement to encode more characters than a 16-bit encoding can accommodate—Unicode 5.0 has some 70,000 Han characters—and the requirement by the Chinese government that software in China support the GB 18030 character set.

[ash2fpga]

Back in win9x days, software like njwin could be used to view cjk text. Who knows if any sort of "overlay" like that would ever be possible.

[jotego]

I think supporting Japanese katakana should be possible. Probably Korean is possible too. We need the MiSTer to be more friendly to the Asian hemisphere.

[zakk4223]

The other problem is that Main Mister is very translation unfriendly; it has a lot of the menu strings embedded directly in the C++ source code with assumptions about formatting baked in (i.e text is aligned via just prepending spaces to strings etc).

That and all the cores have the menu strings baked in as part of the RBF.

Ideally you'd move all the user displayable strings to some sort of a 'resource' file that contained all the translations for a given language. You'd probably want each core to be able to reference a 'private' resource file that could be distributed in the git repo for each core.

If you do it this way users could contribute resource files for a new language without requiring full rebuilds of Main_Mister and every core.

I don't think the work is impossible, it would just be very tedious to convert everything initially.

[dshadoff]

This is a huge subject which implies massive amounts of work... the computer industry took decades to unify (and some would say that it still isn't)...
while this would certainly be a nice thing to have, I wouldn't expect this support soon because of the amount of effort involved.

[grizzly]

From a total noob (but do have some brain left) about pretty much all programing and fpga now.
On top of being a gigantic work, where would/should a resource file/part of the program be able to "run" from?
The fpga or the arm cpu?

If from the fpga does it have enough space left to implement this on every core on the mister?
If from the arm cpu does it have enough clock cycles left so input/network and what ever else is run on the arm side does not suffer?
For example a bit flashier gui according to sorgelig would make input/network suffer some slowdown really fast.

[dshadoff]

It would ultimately affect every part of the system:
- the filesystem (filenames).
- the display subsystem (both FPGA and arm)
- encoding format - there are only a few hundred different ways that these have been encoded over the years, with several thousand characters each - if Unicode were to be used, there would be UTF-8, UTF-16, and other variants to be concerned with. If you implement one, there will be requests for others and various conversions
- font (several tens of thousands of characters) at potentially multiple bitmap resolutions. Again, if you have one, there will be requests for others.
- storage (space on FPGA may be limited for things like this - it would at least double the requirement)
- then there are the localization concerns - menu items would be requested in multiple languages, when they aren't even in multiple roman-character formats
- at some point, there would be requests for top-to-bottom and left-to-right display

It is really endless.

As I said, anybody who has looked at this in the past 30 years knows that it is a huge subject, and even on the systems which currently implement it, it exists due to the work of thousands of people (much of which will not be open-source, or may not apply on this platform, requiring rewrite).

And even on systems which do support it, very few programs are actually written to make use of it. So, there are multiple confusing ways of using 1-byte or 2-byte encoding.

Of course, if somebody is ambitious enough to undertake this, I wouldn't want to stand in their way... but I know enough about what it required, to know that I am not that person.

[aberu]

It is interesting for sure, but given this is an open source project, there isn't anything other than a significant investment in time stopping some clever developer from doing this. A chinese fork of MiSTer would be pretty cool!

[bellwood420]

Filenames:
It would be nice to support displaying cjk characters in filenames.
I have Japanese named snes roms which I dumped from my real carts collection and wish I could use them without converting them to roman characters.

I know it requires tons of work even without support for localization resources.
Just dreaming...

Menu items:
Ideally it would be good to have localized menu.
But I don't think I (Japanese) can get much benefits from localized menu so it may not pay off.

I feel there are not so many words or sentences which are hard to tell on menu.
And as for Japanese, most technical terms are untranslatable. They can simply be replaced fully or partially by Japanese-English in Katakana, which expresses a similar(but weird) sound.
Even converted to Japanese-English, those who don't know well about the feature will not be able to understand it.

From friendliness point of view, it would be better to have things like below in their language rather than to have localized menu, IMHO.
- Help docs
- Commentary articles/videos by enthusiasts
- User community

[FoxbatStargazer]

Wasn't this project founded by a Russian living in Taiwan who chose to make it English centric? I don't know there is much sentiment for translation...

That said I'm sure some way of handling ROM names would be welcome if it could be figured out.

[bellwood420]

I tried and succeeded in displaying Japanese filenames by quick and dirty tweaks.
But I gave it up to complete, although it was an interesting challenge
It is not perfect and universal. There're issues left that I'm not capable to solve
Please don't expect further work...

IMG_0141.jpeg (593.06 KiB) Viewed 5386 times

How I did it?
Filenames given from Linux were UTF8 in my environment.
When executing mount command, I found that /media/fat is mounted with iocharset=utf8.
Maybe that's why but I don't know where it is defined.

Anyway, char* for filename in Main_MiSTer is actually UTF8.
Main_MiSTer seems to be designed to handle one byte UTF8 which equals to ASCII.
So first I added multibyte UTF8 handling to PrintDirectory function in menu.cpp.

Next problem was font.
Luckily I found a free awesome Japanese font(Misaki Gothic) which is designed with 8x8px including 1px margins, the same as MiSTer bitmap font!

Rendering was done by freetype2 library with its Cache Sub-System.
In OsdWriteOffset function in osd.cpp, if the character consists of 2 or more bytes, a bitmap for the character is rendered instead of using original MiSTer bitmap font.

What's the issues?

1. Handling scroll
When filename is too long to fit in 2-lines, it starts scrolling when the menu item is focused.
This logic is on ScrollText function in osd.cpp.
It should also support multibyte UTF8, but it is too complex for me to mod.
Even modding PrintDirectory function in menu.cpp was a nightmare to support expanding to 2-lines for long filename.

2. Universal use
Rendering logic expects font to be designed with 8x8px with 1px margins.
Other font than Misaki Gothic cannot be used properly so it is not universal.

3. Size
Although Misaki Gothic is a great artwork, it is a bit hard to read Kanji(Chinese characters) compressed into 8x8px (actually 7x7px without margins).
10x10 to 12x12 px would be nice but it requires major work on OSD, which I don't want to touch.

4. Unknown bugs
There may be bugs caused by side effects that I'm not aware of.
It's not fully tested.


Changelogs are available in the forked branch below.
Hope someone will improve...
https://github.com/bellwood420/Main_MiS ... p-filename

[jotego]

I tried and succeeded in displaying Japanese filenames by quick and dirty tweaks.
But I gave it up to complete, although it was an interesting challenge
It is not perfect and universal. There're issues left that I'm not capable to solve
Please don't expect further work...

This is already quite awesome.

About translating the cores, well, some time ago most arcade cores moved to the MRA file format so at least we have the DIP switches and button names in that file, which is plain text.

[bellwood420]

About translating the cores, well, some time ago most arcade cores moved to the MRA file format so at least we have the DIP switches and button names in that file, which is plain text.

When it comes to handling MRA files, we need to make changes in support/arcade/mra_loader.cpp where there are a lot of string manipulations.
Low-level C/C++ is not suitable for manipulating multibyte UTF-8.
It will mess the codes up as seen in my changes, make maintaining hard and increase potential bugs

Using fixed width wide character(wchar_t) instead of multibyte UTF-8 would make manipulating easier as it can be used with library functions defined in wchar.h.
However, interfacing with tons of existing functions which expect ordinary char: one byte UTF-8(ASCII) becomes a huge problem with the way.

[redclover1213]

I tried and succeeded in displaying Japanese filenames by quick and dirty tweaks.
But I gave it up to complete, although it was an interesting challenge
It is not perfect and universal. There're issues left that I'm not capable to solve
Please don't expect further work...

I have improved your mod by supporting 8x12px fonts.
It makes Chinese/kanji characters a lot more readable.

IMG_6177.jpeg (2.94 MiB) Viewed 1245 times

IMG_6178.jpeg (4.03 MiB) Viewed 1245 times

My “even dirty hack” is always display one text as two lines on osd.

1. Display upper part of the characters with offset of 4px. It will make sure only 8x4 of the font is displayed on the first line.
2. Display lower part of the characters with a new line in parallel. Full 8x8 px are being rendered as one line.

With 2 lines together, Chinese characters is very readable even on my arcade monitor.
Here is my fork: https://github.com/RickyL1213/MiSTer_8x12_Font

[zhejoe123]

I tried and succeeded in displaying Japanese filenames by quick and dirty tweaks.
But I gave it up to complete, although it was an interesting challenge
It is not perfect and universal. There're issues left that I'm not capable to solve
Please don't expect further work...

I have improved your mod by supporting 8x12px fonts.
It makes Chinese/kanji characters a lot more readable.

My “even dirty hack” is always display one text as two lines on osd.

1. Display upper part of the characters with offset of 4px. It will make sure only 8x4 of the font is displayed on the first line.
2. Display lower part of the characters with a new line in parallel. Full 8x8 px are being rendered as one line.

With 2 lines together, Chinese characters is very readable even on my arcade monitor.
Here is my fork: https://github.com/RickyL1213/MiSTer_8x12_Font

Do you have a layman's version of how to apply the font?