Thursday, June 22, 2017

RetroArch Tone-mapping LUT Shader

Reshade has long had a shader, LUT.fx, that enables users/designers to do tonemapping and other color adjustments without touching any code. Instead, they can do all of their adjustments in an image editing program, such as Photoshop or GIMP, which many people are familiar with already. While my image-adjustment and color-mangler shaders can be used to accomplish those same tasks (Pokefan531 did a great job modifying them to produce his handheld color shaders, after all), they're awkward to work with, since an artist has to make all of his/her changes by twiddling esoteric values in the shader settings menu.

So, I decided to port the Reshade shader to RetroArch so our users could get in on the fun. I ran into some unexpected behavior with the direct port, though, and decided to adapt another similar shader instead. This one ended up having the same weird issue, which I think is related to undefined behavior, but I put a stupid workaround in the shader to mostly deal with it.

Anyway, here's how you use it:

First, take a screenshot that you want to use as your reference (I'm going to use the Sonic the Hedgehog title screen) and then take one of the passthrough palette textures that come with the shader (they're the png files located in reshade/shaders/LUT, named for their color depth). Then, in Photoshop or GIMP or Paint.NET or whatever, open your reference screenshot and paste the palette image down below it:
I find the easiest way to do this is to go to the image menu > canvas size and then anchor it from the top and increase the canvas Y-axis measurement by the height of the palette texture (in my example, I'm using the '16' texture, so I made the image 16 px taller). Paste the palette image in there, move it to the bottom-left corner and then merge the palette layer with the screenshot layer.

Next, do whatever it is you need to do to make the picture look like you want it. That includes brightness/contrast, hue/saturation, indexed color, different lossy colorspace conversions (such as by switching to CMYK colorspace and then back to RGB). I'm going to do a simple hue shift in my example because it's easy to spot:
Once you have it all set, we need to isolate the palette from the screenshot. I think the easiest way to do this is to go back to image > canvas size, anchor from the bottom left and enter the size of the palette (the width of the passthrough palettes is the height squared; I used the 16-bit palette, so it's 16 * 16 = 256 px). Save that image under a new name (I called mine 'hedgehog-palette.png'; if you're using a different palette depth from the default 16, it's probably a good idea to put that into the filename somewhere so you don't forget it) and then drop it into your reshade/shaders/LUT directory with the other palette images.

Now, open the LUT shader preset (cgp/glslp/slangp file) in a text editor and change the line that points to the palette (probably line 7, but YMMV):
SamplerLUT = shaders/LUT/16.png
becomes
SamplerLUT = shaders/LUT/hedgehog-palette.png
Save and exit and then fire up RetroArch, load a core and some content and then load up the shader. It should apply those same color transformations to the game image, like so:
If you used a different bit-depth palette from the default 16, your colors may look crazy and messed up at first, in which case you need to go back into the quick menu > preview shader changes and then change the "LUT Size" runtime parameter to match.

This shader is available from the online updater and/or git in GLSL, Cg and slang shader formats.

5 comments:

Anonymous said...

This is the bomb! I looked at that rgb genesis thread and it help me learn alot about fixing the rgb issue using Genesis 240p test suite game that displays RGB and White bars.

I don't have an account for the forum but as requested by that Natasha user, I made a 99% RGB Fix for both GenesisPlus GX and PicoDrive core:

http://www.mediafire.com/file/5lt6z9ggx42828u/SEGA+Genesis+RGB+Correction.zip

The problem was the Blue Bar lightness at 30% higher than the Red/Green, even though the White Bar showed that the Blues were the same according to eyedropper reader but it wasn't. The Red and Green themselves were way off too which is why some colors looked a bit too blended and eye tearing when looking at Sonic in STH2 while having a bright blue background. The Greens looked a bit "cold" too.

What I did in photoshop was, I decreased the Blue's lightness in the Hue/Sat settings so the highest lightness can be the same as Red/Green/White lightness. Then, I used the Curves setting, selected each bar to lock 'em, and adjusted them to match everything to be like the jEnesis RGB.

Now alls fixed except for 1 white bar that's 1% less Blue. I tired but it seems impossible to have this one intact and there's no real way of fixing it, but this is nothing since it's extremely hard to notice difference vs jEnesis RGB from Test Suite's bars and various other games so it this is a 99% fix.

Many thanks to you mate for the guide and Enjoy!

Hunter K. said...

@Anon
Hey, that's great! Thanks for sharing it. I'll post this over at the forum for you :)

Anonymous said...

Cheers. Heh sorry about the inconvenience but I forgot to add an important note for users who want to fix RGB…

You must Load the default LUT.glslp Preset (16.png) in RetroArch FIRST. Then take a screenshot of the 240p Test Suite RGB Color Bars and then start making adjustments in photoshop.

If default LUT.glslp isn't in use and you took a screenshot of the RGB Bars earlier, you won't be able to get proper results after photoshopping. This is due to RGB value differences between no shader in use vs vanilla 16.png in use.

Oh and of course this has nothing to do with my RGB Fix download itself so no sweat. Hope this helps for those taking an attempt.

MajorPainTheCactus said...

Just came across this article HunterK - very informative! Thanks for taking the time to write it.

Hunter K. said...

Hey man, glad you liked it. As always, let me know if you have any questions or suggestions. It's pretty easy to modify it to access >1 LUT for cycling among different color profiles. Guest's shader does this.

Analytics Tracking Footer