Wednesday, April 5, 2017

Upscaling shaders for pre-rendered backgrounds

In the Playstation 1 / Nintendo 64 era, it was a common practice to pair low-poly 3D models with prerendered photo-realistic 2D backgrounds to give the illusion of a fully 3D-rendered game. This technique was used extensively in the Resident Evil series and Squaresoft RPGs (Final Fantasy 7, 8 and 9 and Chrono Cross), and when we emulate those games, we run into some issues related to the background.

If we use an emulator that supports increased internal resolution, the 3D polygon elements look sharp and crispy but the prerendered scenes are super-pixelated, and the contrast between the two can be jarring and distracting:
An unsettling juxtaposition.
To avoid that, you can stick to the native resolution and let everything look consistently pixelated and then use a full-screen post-processing shader remove some of the rough edges. Unfortunately, many of the algorithms we use for upscaling representational, cartoony pixel art don't always do a good job with the prerendered scenes, as the subtle gradients and natural, random dithering can trip up the pixel comparison and pattern detection methods that work so well in other applications:
Only a few pixels get smoothed; most are skipped
There are some good upscaling algorithms that are designed to deal with this sort of application, though, and we'll take a look at some of the ones available in shader form. We'll be comparing 4x upscales from two native-res captures from Final Fantasy 7:


ScaleFX-Hybrid

Sp00kyFox's scaleFX algorithm does a stellar job on cartoony stuff, and he tweaked the algorithm a bit to better handle small details. It has a rough, stipple texture on edges, unlike the regular version, and maintains details that the regular version smooths away:

It also handles gradients better than the regular scalefx, with reduced posterization.

NNEDI3

This algorithm, which utilizes a "predictor neural network architecture and local neighborhood pre-processing" (whatever that means), was recently ported to RetroArch's slang shader format by a fellow that goes by the name ZironZ. He hardcoded the huge swaths of weights derived from the neural network into the shader passes, which makes for relatively long compile times but respectable speeds once the task is done. He provides a handful of presets, some of which are much too demanding for my modest GPUs, but the basic preset runs in real-time and looks very nice:

This algorithm compares well with the popular waifu2x algorithm, which is also neural net-based but cannot run anywhere close to real-time (these shots are using the 'photo' preset with 'medium' denoising):
waifu2x, photo, medium denoise; no shader form available

waifu2x, photo, medium denoise; no shader form available
While all upscalers have a "signature" in the way their output looks (characteristic swirls or burrs on objects, etc.), much of waifu2x's magic mojo comes from its denoising routine, which gives images a surreal, painterly look when maxed out:
waifu2x, photo, max denoise; no shader form available
waifu2x, photo, max denoise; no shader form available

We can't reproduce that look exactly with real-time shaders, but we can get closer by adding a pre-pass of bilateral blur, which denoises the image by blending neighboring pixels that are close in color/intensity.

Fast-bilateral-NEDI

This combines Xin Li et al's NEDI algorithm (not to be confused with the aforementioned NNEDI3 from tritical) with a fast-bilateral pre-pass to get some of that smooth, surreal look at the cost of lost detail:

The fast-bilateral shader includes a runtime parameter to control the strength of the blending, and I modified it to go all the way up to 2.0 (default value is 0.4 with a max of 1.0), which looks nice but nearly wipes out the tile mosaic:

Fast-bilateral-super-xBR

Hyllian's xBR algorithm is now legendary in emulation circles for its ability to upscale pixel art, and the super-xBR variant is tuned to handle images and photos. The pre-pass of fast-bilateral cleans up some of the images' own built-in noise:

Again, cranking up the bilateral-blur to to 2.0 has an interesting if not useful effect:
And, just for fun/comparison, here's what waifu2x looks like in "artwork" mode (that is, a mode that throws away even more detail) with the denoising at maximum:
waifu2x, artwork, max denoise; no shader available
waifu2x, artwork, max denoise; no shader available
Hyllian's 3D detection shaders

Another alternative strategy involves using an initial shader pass to identify which elements are upscaled 3D models vs which ones are 2D textures. This lets the shader work on HUD elements and backgrounds without getting thrown off by the increased internal resolution:
super-2xbr-3d-6p-smoother (I think) 
jinc2-sharper-3d (I think)

Sunday, March 5, 2017

Raspberry Pi 240p Composite Output

Big news for emulation on Raspberry Pis of any generation: a new firmware update has enabled 240p output through the composite video-out port (a dedicated port on older models and a combo 3.5 mm jack on newer models).

From Raspbian and its derivatives, you can run from a terminal:
sudo rpi-update
and it will automatically install the new firmware. For Lakka, the updated firmware will make its way into the main release at some point, but in the meantime, you'll need to mount your SD/microSD card on another machine and then find the 'boot' directory (for Lakka, this will be located in the ~500 MB partition that holds the system files rather than the bigger partition that holds your games) and make a backup in case anything goes wrong (not likely, but better safe than sorry).

Next, you'll need to download the new firmware files from github (here's a direct link to a zipped download) and copy them into the boot directory, overwriting what's already there. I believe you only really need a subset of the files (bootcode.bin, fixup.dat, fixup_cd.dat, fixup_db.dat, fixup_x.dat, start.elf, start_cd.elf, start_db.elf and start_x.elf), so if it acts strangely, maybe try cutting down to just those files.

In your config.txt, you can use the sdtv_mode directive to change to the new modes (either NTSC or PAL):
sdtv_mode=16    # progressive NTSC
sdtv_mode=0x10  # progressive NTSC (using hex notation)
sdtv_mode=18    # progressive PAL
sdtv_mode=0x12  # progressive PAL (using hex notation)
Also, make sure your config.txt isn't forcing HDMI output via hdmi_force_hotplug=1 (comment it out, if so) and you might want to force composite output, just to be sure, by adding hdmi_ignore_hotplug=1.

Raspbian and derivatives can now also change modes at runtime using the tvservice program:
$ tvservice -c "NTSC 4:3" ; fbset -depth 8 ; fbset -depth 32; tvservice -s 
Powering on SDTV with explicit settings (mode:0 aspect1)state 0x40001 [NTSC 4:3], 720x480 @ 60.00Hz, interlaced
for interlaced mode or:
$ tvservice -c "NTSC 4:3 P" ; fbset -depth 8 ; fbset -depth 32; tvservice -s 
Powering on SDTV with explicit settings (mode: 16 aspect:1)state 0x40001 [NTSC 4:3], 720x480 $ 60.00Hz, progressive
for 240-line non-interlaced mode.

Now, you'll notice that the above line still says 720x480 and applications will still see a 480-line resolution (that is, emulators will want to run at 2x scale). That's because the firmware still acts like normal 720x480 (or 576 lines for PAL) and the composite encoder just displays only half the lines.

This is actually to our benefit because it allows us to easily go back and forth at runtime and has the potential for on-the-fly switching (though RetroArch/Lakka doesn't support that and probably never will). It also allows emulators to show interlaced content without freaking out (that is, a "true" 240p resolution would cut off half of the picture, while this setup will just blank out every other line, which is much less obtrusive).

I have some pics here but, hilariously, the one from the new firmware turned out the worst:
RPi 240p Composite

Analogue Nt - RGB-PPU YPbPr

RPi 480i Composite
If you look closely, you can see that the RPi 240p shot has gaps between each scanline (easier to see on the green waves of the hill than on Luigi), while the 480i shot does not. What the still image can't show is that the 240p image is rock-solid stable, while the 480i image flickers sickeningly. Either way, the RPi composite output obviously does proper NTSC encoding, so blending of dithering and so on happens automatically without needing a shader, which is nice. EDIT: the NTSC encoding is (obviously, in retrospect) using the RPi's full 720 horizontal res timing rather than a console's native timing, so effects that depend on NTSC artifacting may or may not look like you would expect. For example, here's what the "tvpassfail" NTSC test ROM *should* look like:
NTSC Pass/Fail test ROM with maister's NTSC 256 px shader
But this is what the RPi's composite output actually shows:
NTSC Pass/Fail test ROM via RPi composite output
There's still crosstalk, of course, but it's spread across more pixels, which breaks the test. Perhaps unsurprisingly, it looks closer to the 320 px NTSC shader output:
NTSC Pass/Fail test ROM with maister's 320 px shader
With this new firmware, the RPi3 is now arguably the best/easiest/cheapest way to do proper 240p emulation on a typical NTSC standard-def television. Previously, a softmodded Wii was the undisputed champ in this area, as it was the only way to do 240p without putting a series of adapters and converters in the mix. The Wii still has an advantage insofar as it can output a high-quality YPbPr signal (i.e., comparable to the Analogue Nt shot above) but the RPi3 has the ability to emulate more consoles, including great Playstation 1 emulation via PCSX-ReARMed and decent N64 emulation via the core formerly known as GLupeN64 (now renamed to just Mupen64Plus-libretro). The Pi2SCART add-on from ArcadeForge can get a cleaner, "pixel perfect" output from emulators on the RPi, but at ~$50, it costs significantly more than the RPi itself, costs as much as a used Wii and requires adapters to work with standard NTSC televisions. So, in my opinion, RPi is the way to go for cheap/easy "240p" emulation on a regular ol' U.S. TV, while the Wii is a step up in visual quality but more of a hassle (and fewer games/consoles available to emulate). Pi2SCART wins in both respects but also adds significant cost.

Friday, January 20, 2017

Pixel Art Upscaling Test Image

At the RetroArch/libretro forums, we get a lot of requests for shader previews because there are a lot of them and many people--particularly new users--get overwhelmed and don't know where to even start. So, we've batted around the idea for a while but there were some complications, namely that it's hard to get a good representative preview without managing giant, full-res images, and it's difficult to show of the effects of some shaders while still making the previews directly comparable to one another.

To handle these issues, I decided to make a small test image that contains many of the important features for a retro-focused upscaling algorithm to handle (this is enlarged; it's actually 64x64):
It was a fun exercise trying to pack as many tests as possible into this small canvas. There are two gradients--red-to-green and black-to-white--to test banding, several styles of text, some space invaders to test effects on isolated pixels surrounded by another color, the blended/antialiased doom-guy head, a variety of slopes that go from 1:1 up to 6 pixels of run to a single pixel of rise, the pesky circle-C copyright symbol, some parallel lines and a checkerboard to test de/dithering response/false-positives and a multicolored square that turns into a green-and-beige checkerboard in the presence of NTSC signal modeling, like this:
For the preview repo, I ran the test image through each shader preset at 8x integer scale (or slightly above that for the sharp-antialiased shaders). I don't have previews available for all of the presets yet at the time of this writing, but most of the popular ones are covered. These previews will hopefully make their way into RetroArch at some point, but in the meantime you can browse them at the repo.

Thursday, January 19, 2017

Shaders for Sharpest Pixels

As readers of my blog can probably guess, I usually favor scanline and CRT effects on my retro games but there are a lot of folks who prefer sharp-edged pixels. For many of these folks, integer scaling on the Y-axis and 1:1 pixel aspect ratio (PAR) with nearest neighbor (NN) sampling is gospel, but that can lead to weird display aspect ratios (DAR) on many systems, including S/NES, where the PAR is naturally non-square*. This issue is further exacerbated if you want to run your games at 4:3, which is the nominal aspect ratio of CRT displays and is probably most like what people saw playing retro games in their natural habitat.

If you get away from 1:1 PAR/integer with NN sampling, you end up with a lot of problems caused by uncertainty regarding where in the texel (that is, the texture's pixel) the upscaled pixel is actually coming from. This manifests as "shimmering" during what should be smooth scrolling and ugly, uneven pixel sizes on what should be smooth slopes:
The solution for this is to anti-alias the pixel edges, weighting the resulting pixels against their neighbors, and there are a handful of shaders that do precisely this. For my comparison shots, I've zoomed in much further than I usually do and I'm using an NES shot to accentuate the effects. I also limited the interpolation to the X-axis to make it easier to see the effect.

PIXELLATE

Originally written by Fes, this is the OG sharp anti-aliasing shader and has been ported everywhere. It takes four texture samples a small distance from the current pixel and averages them together.
Since the averaging is happening in gamma-adjusted colorspace, it favors dark pixels just a bit, so I added a runtime option to the slang port to do the interpolation in linear colorspace instead:
This avoids the darkening but also leads to "floating" pixels sometimes when an upscaled pixel is flanked by light pixels, as occurs behind/below Mario's ear. So, pros and cons /shrug.

AANN

jimbo1qaz took another stab at the same concept and wareya added some bits to allow for interpolating in "pseudo-perceptual" colorspace. It ends up being slightly darker than pixellate via gamma curve, surprisingly enough:

SHARP-BILINEAR

Themaister took a different approach with this one. It prescales the image to a desired integer scale (I added a default option to automatically choose the largest integer that would fit on the screen) using NN scaling and then use bilinear scaling to go the rest of the fractional scale factor. In this shader, there's no averaging of multiple samples, so the gamma status doesn't matter. It ends up landing somewhere between AANN and pixellate via gamma curve:
This shader is very lightweight because it only samples the texture once and then leverages the GPU's own scaling hardware, which works essentially for free. Note: this effect is the exact same concept as the "prescale" option that appears in a variety of emulators.

There are a few other shaders worth mentioning that I didn't include here, like Inigo Quilez' "improved texture filtering" (we just call it 'quilez' in the repos), which is significantly sharper than plain bilinear scaling while still providing evenly sized pixels, and aliaspider's scaling swiss army knife, GTU, which can produce a similar anti-aliasing effect when pushed to a very high internal resolution.

These shaders are available in Cg, GLSL/slang and quark/GLSL formats.

*This statement is hilariously contentious and causes some people to flip their shit.

Sunday, January 1, 2017

Tung-Sol KT66 Replace 6L6 for HiFi

Size comparison of Tung-Sol KT66s vs a Svetlana 6L6
I've tried a few different varieties of 6L6 power tubes in my Dared VP-20 amplifier, which has been interesting but not particularly exciting. The preamp tubes makes a bigger impact on the sound, in general, and 6L6s all have the same basic sound signature, which is warm and mild, so swapping among the different brands has been more of a null-finding (i.e., still informative, just not much to write/rave about).

I wanted to try some different families of power tubes but hifi amps tend to be less forgiving than guitar amplifiers when it comes to voltage and power-draw changes, so my options were quite limited. Some guitarists replace 6L6s with KT66s, but KT66s pull 1.3 amps of current, which is roughly half-again as much current as a 6L6. However, while researching the KT66 varieties, I found this article which mentioned that Tung-Sol's KT66 happens to pull 0.9A, just like a 6L6!

According to several places online, the Tung-Sol KT66 is so similar to a 6L6 in performance and characteristics as to be "not a real KT66" or "just a 6L6 in a different bottle." However, I found it to sound significantly different from the Svetlana 6L6s I had in my amp before. The KT66s are much brighter, with crisper highs and punchier lows, at the cost of reduced midrange response.

Once I used my preamp's EQ to crank up the mids to be more balanced, I liked the sound signature of the KT66s better than the 6L6s. Comparatively, I would say the difference is akin to toggling an amplifier's "loudness" button on/off. That is, the frequencies that are accentuated by the KT66 sound more lively and the crisp high-end really made my ears perk up. On the other hand, the sub-bass suffered to the extent that I needed to add my long-dormant subwoofer back into the mix to get audible/tactile sub-bass at reasonable volumes.

So, pros and cons, overall. I'm currently digging the change and having fun listening to my favorite albums with the KT66's different coloration but I'm definitely keeping my 6L6s close by for if/when I want to switch back to their warm, balanced tone.

Some other things to keep in mind if you want to try the Tung-Sol KT66 in the place of 6L6s:
1.) Pin 1 of these tubes is tied to the metal base, so be sure to remove any tube clamps that would come in contact with the base to avoid ruining your tubes and/or amp.

2.) These bottles are big af, and so is the base. The glass is about 2" in diameter, so make sure you have enough space between tube sockets to hold them, and if your sockets are recessed, make sure you either have enough clearance for the base or pick up some 8-pin "socket savers" (available for about $10 for 4x from Chinese sellers on eBay), which is what I did.

By the way, you may notice in my pic above that I also picked up some "tube stabilizers" (aka silicone o-rings), which, for the record, made absolutely zero difference in sound quality. They may do something at very high volumes (i.e., when the tubes are getting physically affected by vibrations), but at normal listening volumes, the rings do nothing. Don't bother.

Analytics Tracking Footer