Wednesday, March 9, 2016

Using Rollback to Hide Latency in Emulators

This is something I thought about a pretty long time ago but recently decided to flowchart out in case anyone wanted to implement it.

The way rollback-based netplay (e.g., GGPO and RetroArch's current netplay implementation) works with emulators is that the two players exchange input states (which are small enough to pass back and forth within the time of a single frame of emulation) and then when they diverge, you roll back to a previous known-good time where the input still agreed and then emulate the intervening frames with the corrected input data to catch up (in RetroArch, these are called 'lag frames').

I had an idea for hiding a frame of latency that would work similar to that concept but in a single-player situation (click to embiggen):

The way it works is whenever the player's input changes, you roll back one frame and apply the new inputs retroactively and then emulate two frames to catch back up. This makes your inputs go into effect one frame before you actually pressed the button(s). This wouldn't result in a rollback loop because, even though we feel like we press a lot of buttons all the time when we play a game, most of the time (particularly from the emulator's point of view) we're really just holding a button or two.

You would want the audio to run one frame behind the video, so your audio buffer wouldn't be constantly emptying and skipping around every time you roll back. Instead, it would fill back up with the next frame's audio during catch-up. Despite running a frame behind the video, our brains would unconsciously sync the audio with the video, as they are very forgiving about that sort of thing (this is a known effect).

One drawback is that you would frequently lose a single frame of animation but thankfully our brains are quite good at papering over that sort of thing, as well. The other drawback to this method is that it would require emulating two full frames in the space of a single frame, so the CPU requirements for any emulator using it would be doubled.

Again, this method would hide a single frame of latency. While most PC setups have significantly more than one frame of latency to contend with, every little bit helps.

UPDATE (4/12/2016): I was talking to letoram, author of ArcanFE, the other day and he mentioned that he had implemented something just like this recently and that the lagging audio made everything seem *more* latent, presumably due to the everything-syncs-to-the-slowest effect mentioned in the above link. I didn't ask him whether muting/disabling audio helped with this or not.

UPDATE (4/30/2018): A few months ago, Dwedit--the author of PocketNES and Goomba Color emulators for GBA--mentioned having a similar idea and after some brief planning and discussion, he whipped up a working model and then refined it into a functional feature for RetroArch. It's included in the 1.7.2 release under the name "runahead". Some differences in his approach to the one I described include keeping audio in sync (rather than lagging behind) and having the option of running a second instance of the core to hop over to in the case of audio issues. To avoid weird rollback effects (see: the 'lose a single frame of animation' thing in my original post), Dwedit astutely recognized that as long as you keep the number of rollback frames below the number of internal lag frames in the game (that is, the number of frames it takes for your input to cause a reaction on the screen), the effect is completely invisible to the user.

Friday, March 4, 2016

New and Updated Shaders

It's been awhile since I've done any shader posts, so I figured I'd cover some updates that have happened recently. Same format as usual, the only difference this time around is that instead of zooming into screenshots using photoshop, I made them straight from RetroArch using the 'zoom' from my image adjustment shader. This gives sharper, cleaner detail shots. Anyway...

No Shaders (Nearest Neighbor)

Here is a shot with no shaders for comparison.


ScaleFx

Sp00ky Fox had been working on cleaning up artifacts from the Scalenx shaders and, in the process, worked up a similar algorithm of his own, known as ScaleFx. By cranking it up to 9x, he managed to do some pretty impressive smoothing:

Some notable things here: check out the circle-c copyright symbol, which is very hard to deal with in this sort of shader, as well as the straight lines throughout the logo and on the shallow slope below the dragon coin.

He also made a variant, known as scalefx-hybrid-9x, that uses reverse-antialiasing and creates some interesting depth and shading effects:

It does introduce some haloing, though, and is pretty heavy, performance-wise.

xBR-lv2-accuracy-multipass

For the past year or so, Hyllian has been working to improve his now-famous xBR upscaling algorithm, mostly by pairing it with other algorithms and working to improve handling of problematic edge-cases. Very recently, though, he changed the way corners are detected and added a new color diff algorithm, which fixes some weird artifacts that could occur when bright red and bright blue pixels were next to each other (artifacts not pictured):

Here is an older version of the same shader for comparison:

Again, the copyright symbol is a good example of the improvements, along with the circles inside the number 9s. The updated version also works quite well with Playstation-era antialiased images, while ScaleFx works best on bold, cartoony graphics, like Super Mario World and Shantae.

CRT-Lottes Updated

Since we first ported Timothy Lottes' scaling pixel art shadertoy, he did a couple of iterations to add bloom and a few more variations on his awesome shadow mask code. r5 incorporated those updates into our port and threw in some runtime parameters (including the aforementioned mask variations):

This is the default compressed TV-style shadow mask^^.

This is the Trinitron-style aperture grille^^.

This is the stretched VGA-style shadow mask that was used in the original shader^^.

And this is another VGA-style mask with larger phosphors^^.

CRT-Royale-Kurozumi Preset

Shmups user Kurozumi posted some really nice settings for TroggleMonkey's CRT-Royale shader that makes it look very much like a high-res broadcast monitor (e.g., Sony's BVM line or the 800-line PVMs):

I put these settings into a cgp preset, located in the 'cgp' subdirectory of the main common-shaders repo.

Analytics Tracking Footer