Lua palette quantization

[warmCabin]

Don't mind me, just using your appveyor to avoid setting up Visual Studio on my new PC :)

[warmCabin]

Definitely seeing some improvements by switching away from the green channel rods n' cones biasing thing. When I completely turn off the color caching and run this test script, there is a noticeable performance hit when turboing. I'd say it goes from 25x to 10x speed. As such, I changed the color cache to 5 bits per channel and it seems fine... But I'm still a bit leery, so I might make a Lua API that can turn this on and off just in case 5 bits still makes things weird.

Here are some comparison results. Hopefully it's not too hard to see. With the green bias, the ghosts get kind of yellowy when they stack up.

3-bit green bias (the ways things are in master)

3bit-green_bias.mp4

no cache euclid (best results, noticeable performance dip when turboing)

no_cache-euclid.mp4

5-bit Euclid

5bit-euclid.mp4

3 bit euclid

3bit-euclid.mp4

no cache green bias

no_cache-green_bias.mp4

[warmCabin]

Alright, finally got around to adding some Lua stuff.

gui.clearcolorcache

I'm adding this just in case anyone (namely me) still finds that the colors look a little funny. You can even spam it every frame if you want to. Hopefully we won't need it with the larger color cache.

gui.setcolormatchformula

Something I added for testing purposes. I will almost certainly revert this unless you find it valuable.
The blending modes are:

• Original
• Original, but with delta squared (looks better)
• Basic Euclidian
• "Redmean," an algorithm I stumbled upon in my... OK fine, Gemini recommended it.

Intended to be paired with this new version of my little palette tester script:

colorblending.lua

local formulaNames = {"Luminance squared", "Euclidian", "Redmean"}
formulaNames[0] = "Luminance original"
local formula = 3
local opacity = 0.7
local fullOverride = false

local prevInp = {}

gui.register(function()

    inp = input.get()
    if inp.right and not prevInp.right then
        formula = (formula + 1) % 4
        gui.setcolormatchformula(formula)
        gui.clearcolorcache()
    elseif inp.left and not prevInp.left then
        formula = (formula - 1) % 4
        gui.setcolormatchformula(formula)
        gui.clearcolorcache()
    end
    if inp.up and not prevInp.up then
        opacity = math.min(opacity + 0.1, 1.0)
        gui.clearcolorcache()
    elseif inp.down and not prevInp.down then
        opacity = math.max(opacity - 0.1, 0.0)
        gui.clearcolorcache()
    elseif inp.home and not prevInp.home then
        fullOverride = not fullOverride
        gui.clearcolorcache()
    end
    prevInp = inp

    local centerX, centerY = 64, 100
    local baseX, baseY = centerX + 60 * math.sin(emu.framecount() / 60), centerY + 80 * math.cos(emu.framecount() / 80)
    local size = 8

    gui.opacity(fullOverride and 1.0 or opacity)
    for i = 0, 0x3F do
        local x, y = baseX + (i % 16) * size, baseY + math.floor(i / 16) * size
        gui.box(x, y, x + size + 1, y + size + 1, string.format("P%02X", i), "clear")
    end
    
    gui.opacity(1)
    gui.text(10, 10, formulaNames[formula])
    gui.text(10, 20, string.format("%.1f", opacity))

end)

print "Keyboard Controls:"
print "  left/right: cycle blending mode"
print "  up/down:    change opacity"
print "  home:       toggle full opacity override"

A few screenshots

Original	Euclidian	Redmean

---

And to give you an idea of what the 3-bit to 5-bit cache update is doing: Each grid region is considered the same color. I changed it from the big squares to the small squares. Kinda looks like that Hues & Cues game I keep seeing at Target!

[warmCabin]

Looks like you guys have been busy with the Mac build scripts. I'll go ahead and rebase.

[warmCabin]

@zeromus @bbbradsmith
Can you guys take a look at this?

[zeromus]

why does the cache get wacky? I can't imagine why something like that would happen. if there's an explanation for it I might accept it, but otherwise it seems like a bug-in-waiting.

that would seem to be the only blemish on here (I don't mind keeping the algorithm picker for old-video compatibility or better blazing performance)

[warmCabin]

The cache gets wacky because the colorspace regions are too broad. It stores the first color it happened to encounter in any given region, so over time, you end up with some strange mismatches. By changing it to use the upper 5 bits, the regions get tightened up and it's still just as fast. That's what I was trying to convey with those pretty rainbow squares.

Since the same problem is technically still here (just less obvious), things might still look a little off. That's why I wanted to make gui.clearcolorcache.

[zeromus]

I don't understand. The goal here is to figure out which palette entries to use when drawing translucent stuff because for some godforsaken reason we're still doing lua drawing in a layer with paletted colorspace?

[warmCabin]

Right. I wouldn't even know where to start rewriting that... I guess you could pass around an extra 32-bit buffer through FCEUD_Update and friends, and do alpha blending in the blit code?

---

How about some shitty paint.net diagrams?
Let's say A would match to A', and B would match to B'.

But if we happened to encounter A first, then this whole region would be cached to A', which would definitely not look right no matter what matching formula we used.

[zeromus]

The presumption is that A' and B' are both "not off by more than 1" good approximations for both A and B. It's implicit in the binning design. This sounds like normal situation, not a defect. If the results aren't visually appealing then the parameters are too coarse. IMO it's an impossible problem so one result or another will always be visually unappealing. This is what dithering is for, though I don't think it's desirable output in this case.

I think I would rather see a lua api that sets the coarseness (and hence memory usage and somewhat the performance implications both at runtime and startup) rather than something to rejigger the cache because I just can't see how anyone would use the latter in practice.

As I'm sure you know by now, it would be better to fix the junky old code so it doesn't need to mix and output in NES indexed-color space.... I would hazard a guess, the only reason this has never been done is that it wasn't important enough to anyone to have sensibly-colored lua overlays. As long as it was fine as junk then the junky results were fine. If you need something artistically beautiful and smooth, then the time has come to smash the whole presentation pipeline

[warmCabin]

If the results aren't visually appealing then the parameters are too coarse.

I'm with you. I believe I've fixed that by setting it to 5 bits per channel.

I think I would rather see a lua api that sets the coarseness ... rather than something to rejigger the cache ...

Sure. It should probably also have a mode to disable caching entirely and take that performance hit, which is what I was really trying to achieve with gui.clearcolorcache.

If you need something artistically beautiful and smooth, then the time has come to smash the whole presentation pipeline

Yeah... No way I could achieve something like that on my own, though. At one point I saw you guys had plans to migrate to DX9; I figure it could be a part of that initiative.