[we need help!] Not possible to map certain characters for custom font

[tangar]

charmap.exe - to open windows fonts..

=============================
UPDATE:
2 and 195 glyphs apparently works! So we need to fix 22 glyphs, not 24!
=============================
UPDATE #2:
Actually it's not possible to use 129, 141, 143, 144, 157 glyphs cause they are reserved by Windows. But it's still possible to fix 16—31 glyphs!

It's already knows for 1+ year ( viewtopic.php?f=9&t=831#p3763 ) but it still not in the guide. Of course devs knows about this problem, but I wanna bring more attention to this critical bug (not that it's missing in the guide, but that we still can't map there any characters). It's deserve separate topic.

The bug:
Currently it's not possible to map characters at:
0, 16—31, 129, 141, 143, 144, 157 (more info there - https://www.tomenet.eu/guide.php?chapte ... o%2027#r49 , but as I said in the beginning, this article is not quite accurate)

Why do we critically need this fixed - cause it would give us extra _22_ symbol to use in graphical font: https://www.tomenet.eu/phpBB3/viewtopic.php?f=15&t=1149 22 symbol would make huge difference.

I beg every person who know something about coding - please help to investigate this bug, why it's broken - currently it's seems nobody got even a clue. It's very important to solve this issue.

Source files for the game could be found there: https://tomenet.eu/downloads.php

Please don't hesitate to post any assumptions and hypothesis considering this bug, lets discuss it.

p.s.
If it wouldn't be possible to find this bug by our forces, maybe we could:
1) make crowdfunding for this 'project' - create a prize fund ($) and give it to the person who would find how to fix this bug
2) post this issue as 'one-time-job' to freelancer.com or other 'job-hunting' web-sites - and try to find person who could do it.
3) make Tangar learn how to code and fix this bug ('mission impossible' ).
Please post any other suggestions too.

====

post @ reddit: https://www.reddit.com/r/roguelikedev/c ... d_a_bug_d/

[tangar]

So far we got several ideas:

LoSboccacc

I don’t remember the code point off my head but the pattern smells of utf-8 treated as ascii and the holes > 128 might match the multibyte codepoint, but I don’t have the tables around

PowerWyrm

Hmm... that's too random to be anything in the code. I guess it comes from your font. Are these symbols in your font mapped to non-ASCII characters? Did you start from the original font file? A modified file?

My answer to PowerWyrm:
Previously dev's already fixed similar bug, when it was possible to map only 128 or so symbols... and now there is problem only with other symbols, like 16-31 symbols, so it still could be a bug in code.
I've modified original file (resized it).

mushroom patch

The numbers you list mostly correspond to control characters or other non-printable ASCII/extended ASCII codes. I would speculate that the numbers you are using to identify tiles are being fed through something that interprets them as codes of that type.

Widmo

Hi Tangar.

A nice tileset you have there. At a similarly nice license, I see. You deserve some help for sure and I suppose looking at the issue myself for a bit could not hurt.

More information would be appreciated. In linked posts you mention developer knows about the problem but I could not find any notes on the topic. Is it really that hard nut to crack or nobody could spare the time to investigate the matter?

Meanwhile TomeNET is now compiled locally on my Arch Linux. It is up and running. During loading the game complains your prf files are buggy in two ways (tested on 16x24). First, there is line announcing the name but it is not defining anything. Maybe you can just comment it out. Second, F:218 entry lacks a second colon and is probably ignored.

Could you please tell us how the failure exactly looks? I edited remapping of up staircase to 144 and got a thin +. 195 gets me Ã, 20 looks like a bursting O, 16 like majestic I, 2 gets me roughly this ▓, 157 seems empty, 17 has some waves. On the other hand zero is promptly ignored as if nothing was defined as remapping. Original symbol is displayed.

If on Windows you get no remapping behavior for the whole range you specified in opening post then the problem source is Windows specific, likely lying somewhere in main-win.c. Index zero is unlikely to be ever recovered as valid mapping since if I understood server code correctly it is used to denote absence of remapping.

Also, if you could get me bcf or pcf file for 14x20 size of your font that would help me somewhat. The 16x24t.pcf for which you provide download is too big for my display. I am unable to convert fon files because currently I have neither Windows nor wine.

My answer:
Hi ho, Widmo!
Thank you very much for so interesting feedback. I'll add it to our headquarters ( viewtopic.php?f=9&t=1737 )
I suppose it's really hard bug, because dev's fixed similar bug in the past, but this particular one is unsolved for a long time and dev's got not idea what's the problem atm
Considering how errors looks.. I'll make an investigation and would post a report at headquarters.
there is 14x20:
http://mmoforum.org/images/games/tomene ... 1-8/fon.7z
http://mmoforum.org/images/games/tomene ... 1-8/prf.7z
Thank you very much for your time

stevenportzer

What happens if you try to map to those characters? Based on the problematic characters, my guess is that something's interpreting them as control codes or something like that for some reason, but I don't think that narrows down the source of the problem very much.

My reply:
it's becomes blank AFAIR, but I'll check it and post in headquarters soon!

[tangar]

Investigation into the problem (video):
https://www.youtube.com/watch?v=MDYDr2Ey5nY

List of problem glyphs:

0 linked to 35 (#)
2 WORKS!

16 ┼ 253C 197
17 ◄ 25C4 17
18 ↕ 2195 18
19 ‼ 203C 19
20 ¶ 00B6 20
21 ┴ 2534 193
22 ┬ 252C 194
23 ┤ 2524 180
24 ↑ 2191 24
25 ├ 251C 195
26 → 2192 26
27 ← 2190 27
28 NULL
29 NULL
30 NULL
31 linked to 35 (#)

129 NULL
141 NULL
143 NULL
144 NULL
157 NULL
195 WORKS!

List of code page 437 table:
https://en.wikipedia.org/wiki/Code_page_437

TomeNET source could be found:
https://www.tomenet.eu/downloads.php

[PowerWyrm]

Since I have a bit of time, let's do some testing...

- downloaded TomeNET on my laptop
- ran the game to get an INI file
- main window mapped to: .\lib\xtra\font\8X13.FON

Let's try with that... So I open the guide and go to section 2.5a for a tutorial on custom fonts.

- creating file named font-custom-8X13.prf in \lib\user
- adding entry F:193:6/126 to remap brick roofs to blue '~'
- relaunching the game -- nothing changed (?)
- entering font menu by pressing = and f, then changing font, relogging
- entering font menu by pressing = and f, then changing font back to 8X13, relogging
- still no change (!)

Is this feature actually working at all???

Ok second try...

- copy font 8x13x.fon from Angband 4.1.2
- rename font-custom-8X13.prf to font-custom-8x13x.prf
- relaunching the game
- entering font menu by pressing = and f, then changing font to 8x13x, relogging
- still no change (!!)

Hmm maybe I'm modifying the wrong feature, let's just remap 190-195 to blue tilde... Nope, brick roofs are still red #. Am I missing something? Let's try with a premade font, let's say 10X16TG... and it works, so it must be something else.

Ok, other prf files start with the line "font-custom-# File: font-win.prf", maybe I need to add that line in my prf file. Let's see... nope, again I get red # and my modifications are lost. It seems the prf file is never loaded.

Ok another try... this time with font file named 8X13X.FON and font-custom-8X13X.prf... aaaand still nothing. The file is simply not loaded...

[tangar]

PowerWyrm, thank you very much for participation in this project! I've recorded video to show how to rearrange symbols in font:

https://youtu.be/mv7bxDMKYLo

(this isn't public video, access only by link)

[PowerWyrm]

Ok I think I know what I did wrong...

I seem to recall that TomeNET directory is copied elsewhere and files are loaded from that directory, so modifying fonts and prf files in the install directory does nothing. Let me check... bingo! That's it... there is a copy of the /user file in c:/users/current_user/TomeNET-user!!!

So I need to copy font-custom-8X13X.prf in that directory! I don't know how it works for you, because in the vid you modify /lib/user/font-custom-8X13X.prf, but for me that doesn't work, I have to modify c:/users/current_user/TomeNET-user/font-custom-8X13X.prf.

So let's try this again:
- copy 8x13x.fon from Angband to TomeNET (this is the 8x13 font with 255 characters instead of 127)
- make font-custom-8X13X.prf in /lib/user with one line: F:193:6/126
- !!! COPY THE FILE IN c:/users/current_user/TomeNET-user !!!
- set Font=.\lib\xtra\font\8X13X.FON in TomeNET.ini
- relaunch the game
- boom! it works! brick roofs are now a blue tilde!

[PowerWyrm]

Ok now using the 8X13X font to test mappings...

Let's try first with symbols 127, 128, 129...

- 127: gives me an ugly square, that's not what I see when I edit with Fony (I should have some kind of wall)
- 128: this should give me a blank... but I get the € symbol instead
- 129: this should give me a blank... and I get a blank
- 130: this should give me a blank... but I get the , symbol instead
- 131: this should give me a blank... but I get the ƒ symbol instead

Ok let's edit the font with Fony and put A/B/C/D at position 127/128/129/130...

Ok, I still get the ugly square, euro, comma and f symbols, so that's clearly a problem with Fony. I have an idea here...

Now let's open the character table from Windows: Start menu -> Windows Accessories -> Character table, select any TT font (for ex: Arial), check "advanced display", select "Unicode". What do we have here? Symbols start with '!' (0x21) to '~' (0x7e) and then restart with reversed '!' (0xA0 == 161) up to 'ÿ' (0xFF == 255). When I look at what I have in Fony... it's the same!!!

Now let's change the character set in the character table from "Unicode" to "Windows: Western" and see what this changes... SURPRISE! Characters from '!' (0x21) to '~' (0x7e) are unchanged... but starting from 126 we have the following:

- 127: nothing
- 128: the euro symbol!
- 129: nothing (doesn't exist)
- 130: comma!
- 131: f symbol!

And what are the symbols that are missing in the table? 129, 141, 143, 144, 157, 173. Exactly the symbols you cannot remap...

I therefore think I've found the problem! You cannot remap the symbols in Windows... simply because they don't exist! The game clearly uses "Windows: Western" character set when displaying symbols, but you try to edit the font using an Unicode editor. In Fony, the euro symbol doesn't show up at position 128 because in Unicode, the euro symbol is at position 20AC (coded on a double byte). Anything you put at position 128 is ignored. At this point, I don't have any alternative to Fony, so I cannot test editing fonts with the western character set, but I'm sure if you do that you won't have any problem.

The funny part is... if you select "Windows: Cyrillic", the character set has no holes (129, 141 and co are mapped to 'Ѓ', 'Ќ' and other cyrillic characters which are perfectly fine), so you could map whatever you want... if TomeNET was translated in Russian.

[tangar]

Ok I think I know what I did wrong...

I seem to recall that TomeNET directory is copied elsewhere and files are loaded from that directory, so modifying fonts and prf files in the install directory does nothing. Let me check... bingo! That's it... there is a copy of the /user file in c:/users/current_user/TomeNET-user!!!

So I need to copy font-custom-8X13X.prf in that directory! I don't know how it works for you, because in the vid you modify /lib/user/font-custom-8X13X.prf, but for me that doesn't work, I have to modify c:/users/current_user/TomeNET-user/font-custom-8X13X.prf.

To make it works in TomeNET directory (as I showed in my videos), you could change TomeNET config file:
[Base]
LibPath=.\lib
; do not move lib\scpt, lib\user and sound/music live configs to the user's
; system home folder but keep using the folder where TomeNET is installed.
DontMoveUser=1

Ok now using the 8X13X font to test mappings...

Let's try first with symbols 127, 128, 129...

- 127: gives me an ugly square, that's not what I see when I edit with Fony (I should have some kind of wall)
- 128: this should give me a blank... but I get the € symbol instead
- 129: this should give me a blank... and I get a blank
- 130: this should give me a blank... but I get the , symbol instead
- 131: this should give me a blank... but I get the ƒ symbol instead

Ok let's edit the font with Fony and put A/B/C/D at position 127/128/129/130...

Ok, I still get the ugly square, euro, comma and f symbols, so that's clearly a problem with Fony. I have an idea here...

Hm.. Sorry, maybe I misunderstood something, but I don't see problem with Fony. Actually when I open 8X13.FON in Fony, there is only 127 symbols (you could see it https://youtu.be/mv7bxDMKYLo ), and it's not possible to add 128, 129, 130, cause they are LOCKED. Last, 127 symbol is 'ugly square'. This font got 'Last char' property = 127. So when you assign in-game values to non-existing glyphs spots it's surely take them from other place.

So if you would change any .FON files from my custom font (where there is 255 symbols), it's totally ok to remap 127/128/.../130 , it works perfectly, except 129, which is bugged.

I therefore think I've found the problem! You cannot remap the symbols in Windows... simply because they don't exist! The game clearly uses "Windows: Western" character set when displaying symbols, but you try to edit the font using an Unicode editor. In Fony, the euro symbol doesn't show up at position 128 because in Unicode, the euro symbol is at position 20AC (coded on a double byte). Anything you put at position 128 is ignored. At this point, I don't have any alternative to Fony, so I cannot test editing fonts with the western character set, but I'm sure if you do that you won't have any problem.

There is no problem with 128' symbol, it's works ok. You got problem cause you used font with max. 127 symbols in it, not 255. We got
problem with 129 - please check my previous message to see list of bugged symbols

[PowerWyrm]

For my test I used 8x13x font, with is the extended 8x13 font with 255 symbols, not the one with 127 symbols. This font is not packaged with TomeNET, I copied it from the /lib/font directory of vanilla Angband.

When editing with Fony, you see symbols up to 127, then you have a blank area from 128 to 160 then it restarts at 161 with '¡' up to 255 which is 'ÿ'. This is because Fony is coded in Unicode language and uses the Unicode character set to display symbols. You can verify this by opening your Windows Accessories, choose Character Sets, select a font like Arial and the Unicode character set.

To actually properly edit a font for TomeNET, you would have to use an editor that could deal with Windows Occidental character sets, not just Unicode. Then instead of the above list of symbols, you would have symbols up to 127, then € , ƒ „ … † ‡ and so on from 128 to 160, then the same character again between 151 and 255. The characters between 128 and 160 are not Unicode. You can verify this by opening your Windows Accessories, choose Character Sets, select a font like Arial and the Windows Occidental character set.

Tomorrow I'll take the look at the source code itself, in particular the Term_text_win() and similar in main-win.c, to see how TomeNET handles text. If the problem is just an ANSI/Unicode conversion, it should be possible to fix the problem programmatically.

[PowerWyrm]

Ok, looking at the code it's actually an Unicode/ANSI problem. TomeNET uses the old Angband code that is using "char" to display text:

Code: Select all

static errr Term_text_win(int x, int y, int n, byte a, const char *s)
...
ExtTextOut(hdc, rc.left, rc.top, ETO_OPAQUE | ETO_CLIPPED, &rc, s, n, NULL);

The new Angband code is using wide chars to display text in Unicode format:

Code: Select all

static errr Term_text_win(int x, int y, int n, int a, const wchar_t *s)
...
ExtTextOutW(hdc, rc.left, rc.top, ETO_OPAQUE | ETO_CLIPPED, &rc, s, n, NULL);

See also:
- the Term_fresh_row_text() and co methods, which are using the text hook above to display symbols
- the Term_draw() method, which actually maps a symbol at its position on screen

If you want to use Fony and display text using Unicode charsets, you have to adapt the code as it was done in Angband. But yeah... that's a *HUGE* change, as it almost forces you to replace all occurrences of "char" by wide chars and do ANSI to Unicode conversions at many places in the code. The work has been done in Angband and committed in Git repository, so you can see the amount of changes this would require by just looking at the commit log. For example:

https://github.com/angband/angband/comm ... ccd706b96b