xsession-errors Log Filling Hard Drive Ubuntu

Last night, all of a sudden, I started getting poor performance out of my HTPC running Ubuntu 12.10 Quantal Quetzal, followed by a complaint that my /home partition was running low on space and only had ~350 MB of free space. A minute or so later, I got another warning that only ~35 MB were free! :O

I clicked the hard drive disk usage analyzation utility to figure out what was up, but it showed my largest directory as weighing-in at just ~29 GB.

Turns out the culprit was a gigantic, hidden file called .xsession-errors, which had ballooned to 170 GB over the course of a couple of hours, apparently. You can check if this is your problem by typing this into a terminal:
ls -a -s | grep xsession
This file is archived at every reboot, with the previous version renamed to .xsession-errors.old and a new log created in its place. If this file is the source of your troubles, you'll want to see what's been spamming your system with errors, but, obviously, such a huge file isn't going to open gracefully in a text editor. However, we can use the 'tail' command to check out the last few lines, like this:
tail -10 .xession-errors
That should give you an idea of what's causing your problems. Once you've had a good look, go ahead and delete the file and then reboot your system to free up the space:
rm .xession-errors* ; sudo reboot now
As soon as your system comes back up, look for the error log again using the first command (and then a third time to see if the size changed in between commands) and then open it in a text editor to see what's up:
gedit .xsession-errors
Scroll down to the bottom and see if you spot any errors repeating. In my case, there was a constant stream of errors from zeitgeist-daemon not being able to find file:///whatever/file.path. Zeitgeist is a program that keeps track of pretty much everything you do on your machine and uses that data to populate 'recent files' lists and so on. It also adds overhead to your system and occasionally shits the bed by making giant error logs ;)

If you don't mind getting rid of the Unity desktop, you can just flat-out remove it and its associated packages like this:
sudo dpkg -r zeitgeist*
However, if this is too invasive for you, you can use the following command to remove the read and write attributes from the zeitgeist-daemon database:
chmod -rw ~/.local/share/zeitgeist/activity.sqlite
You can test the effect out by typing:
zeitgeist-daemon --replace
It should error out with a complaint about not being able to access the database, which is exactly what we want. This essentially cripples the daemon without risking breaking any other packages, present or future.

Go about your normal tasks and make sure everything still works like it should (anything that used zeitgeist will obviously be broken, but I mean everything that you normally depend on). If all is well, you're finished. Pat yourself on the back.

UPDATE: The problem popped up again, even with zeitgeist removed, so I tailed the error log again and saw this line repeated over and over:
Authentication deferred - ignoring client message
This error appears to be a result of the vino server (Ubuntu's built-in VNC server) getting hammered with bruce-force login attempts. It appears that the server allows a few bad attempts and then ignores the rest, which is good. The attempts still get logged, though, so I thought it best just to remove vino altogether, since I never use it anyway:
sudo apt-get remove vino

Designing a Large-scale Phosphor Filter/Shader

Update (5/9/2013): I ported my shader to the Cg shader language and made some improvements in the process (click here to download). It now looks decent at sane, non-4k scale factors and has some easily-modified variables to allow users to make their own adjustments more easily. See the bottom of the post for new screenshots.

Update (3/29/2013): Some other versions of the shader and more screenies at the bottom of the post.
Update (3/18/2013): I think the shader is pretty much done! You can download it here. Skip to the bottom of the post for images and some details.

With 4k TVs on the horizon and high-dpi computer monitors already here, I figured it was time to start thinking about using these screens to better recreate the look of CRT TVs.

cgwg's CRT shader is already fantastic, but it was designed with the assumption of relatively low resolution (i.e., the current HD/1080p paradigm at the upper end and 720p screens at the lower end, resulting in ~3-4x scale factors) and single-pass shader implementations, and it makes certain concessions with those limitations in mind.

At 10x scale (2240 vertical pixels for NTSC) and assuming multipass support, we can forego some of those concessions, though. For example, you can draw the shadow mask right on the screen at 1 pixel width and each phosphor lens at 3 pixels tall by 9 pixels wide, which gives you a pattern like this (blurriness comes from the bilinear upscaling; click for normal size):

In this figure, the phosphors have a horizontal orientation rather than the vertical orientation seen in all of the phosphor images I've found online (like this one). I tried both ways and the vertical orientation didn't look right to me, insofar as it had vertical scanlines rather than horizontal ones. I'll show results both ways later on and you can judge for yourself.

Update (3/18/2013): Turns out the vertical orientation is correct. I got a macro lens and took some pictures of a CRT up close (the letter 'e' followed by a heart icon from Zelda OoT):

I still think it looks better horizontally at small scales (i.e., 1080p and lower) but vertically at larger scales. Anyway, back to the original post.

With that LUT tiled up to cover the entire SNES image (you can download it here), we're ready to add in our image, scaled to 10x using nearest-neighbor, which looks like this (click thumbnails for full resolution):

Next, we need to invert the colors in the image, which looks like this:

We then need to subtract these pixel color values from our LUT matrix, which results in this:

We subtract because the phosphor LUT represents a full 100% brightness in all phosphors, which is blended by our eyes into flat white. In our image, the white pixels are inverted to black, which subtracts nothing from our LUT and leaves those parts white, while magenta blanks out the red and blue pixels, leaving just green, and so on. Likewise, brighter pixels are inverted to dim, and they subtract less brightness from our LUT phosphors, leaving them bright. At this point, we have our basic phosphor coloration and luminance values represented.

Next, we'll make a new pass using the result of the previous pass as input (or in Photoshop, create a new layer), to which we'll apply a gaussian blur that will approximate the luminance-based color bleed into neighboring phosphors (no picture here, since it looks the same, just blurry; not very exciting). According to cgwg's comments in his existing CRT shader, this blur portion may be better suited to a 2-part process of blurring, once horizontally and once vertically, but I just did one all-over blur. You can use a blur intensity of 3 pixels to keep some sharpness or increase the value to 5 pixels to get a more bloomy, halated look. I'll show examples of both in a bit.

Obviously, this is image is still waaaay too dark, so we'll also need to crank up the brightness of our blurred layer to around 150% and combine it with the underlying raw image using the 'screen' method, which lightens everything a bit more. The math for a screen combine is f(a , b) = 1- (1 - a)(1 - b), where 'a' is the base layer value and 'b' is the blend value.

That gives us this result:

At this point, it looks pretty good and we could probably call it quits, but the darkening from the shadow mask is still monkeying with our perceived colors a bit, so I played around with the levels to try and get something that looked closer to the original image. I found that moving the black point to around 23 and the white point to around 179 while keeping the gray point at 1.0 gives a good, bright result:

This adjustment also ensures that blacks are actually black and not brown and whites are white instead of gray. Here's a comparison shot:

While we're looking at this Super Metroid shot anyway, lets zoom in and take a better look at the luminance-based glow created by our blur:

You can see where the brighter pixels bleed into the adjacent phosphors, particularly on the glowing suit highlights.

Using a 5 px gaussian blur bleeds the colors even more but ends up making the entire picture more hazy (3 px blur followed by 5 px blur):

You can see from the thumbnails that the 5 px picture looks nice from a distance, while the 3 px looks a lot better up close. So, you could make a valid argument for either one depending on your preferences and intended viewing distance.

The above process could work alright at a 5x scale, using a bilinear reduction on the LUT and a 1.5 px blur, and not look too bad:

But dropping it to 4x starts to really cause some problems with the colors, giving the appearance of too much gamma: 

At these scales and lower, cgwg's CRT shader looks much, much better, due to his compensation for LCD subpixel aliasing (among other things, such as gamma interpolation, screen curvature, etc.).


The last thing I want to cover is the phosphor orientation. I oriented my LUT with red at the bottom and blue at the top because doing it the opposite way looks weird up close (notice how the red pixels on the shell create a sort of halo instead of having the intended cartoony black outline, as compared with the pictures above):

Likewise, using vertical phosphors--as shown in all of the phosphor images I could find online--looks like crap close up *and* far off (eschew the cutoff edge; click to embiggen):

At the moment, I've just been playing around with Photoshop, but I hope to do some work on a legit shader that performs this process in real time. Unfortunately, I suck ass at writing GLSL, so it may never get anywhere. If anyone else would like to take a stab at it, I'm happy to provide any information I have.

Update: For the Photoshop steps, Romain Dura has helpfully converted the math functions to GLSL, and the gaussian blur steps from cgwg's halated CRT shader can drop right in.

Update 2: I cobbled together some code that works at least partially and I'm pleased with the outcome so far. I don't have a high dpi screen, so I'm stuck working at 4x, but I think the code should work fine with a larger resolution.

Currently, I can get it to the point where it has the shadow mask and phosphor lenses, as well as gaussian blur (though the blur is happening underneath the phosphors, which isn't exactly what I wanted):

This is obviously much too dark, but when I tried to lighten it up, I couldn't get things looking quite right. It would either blast out the colors underneath or make the phosphors visible on a flat black screen, which is unacceptable. When I tried adding a bloom pass, the phosphor bleed looked good (if a bit exaggerated) and the brightness was pretty close but the colors got totally wrecked >.<

Oh well... I'll keep at it. If anyone wants to play around with the code I've got so far and/or play around with my phosphor LUT, you can get it here.

Update 3: As cgwg pointed out in the comments, mapping the phosphors 1:1 on the SNES pixels, like I did above, isn't really how TVs worked. They were designed to receive a 480i signal (i.e., 2 sets of 240 lines of resolution, alternating which lines get displayed for an effective 30 fps), as per the NTSC broadcast spec, which Nintendo and others tweaked to show just 240(ish) lines, non-alternating, at 60 fps. This format goes by many names, including non-interlaced mode, 240p (not a real spec, btw) and "double-strike."

I did some more mockups using the same procedure as above, but using a phosphor array that's ~480 pixels tall (I actually went with 448; i.e., double the SNES res) and started with an image that should match the "double-strike" idea at 10x, namely, a 20x image with half of the pixels blanked out, like this (note, it looks exactly like if you added a 100% black scanline filter; if this is somehow incorrect, I'd love to hear about it in the comments):

Since the vertical resolution doubled, I also doubled the pixel radius of the gaussian blur, up to 6 px and, instead of adjusting the levels, I increased the brightness by 300% and the contrast by 25%, which resulted in this (crap: blogspot won't display that large of a file and insists on shrinking it and converting to jpg; here's a link to download the full-res if anyone wants it). And here it is shrunken bilinearly back down to 10x: 

 It still looks pretty nice. And here's a detail shot:

Interestingly, at this scale, the vertically oriented phosphors look a lot better and produce an image that's much less scanliney, so perhaps it is the better option:

Here's a link to download that vertically oriented shadow mask / phosphor LUT, if anyone's interested in playing around with it.

UPDATE (3/18/2013): After digging around in an old thread on byuu's forum, I came across a post from Themaister where he explained how to go about using and accessing various framebuffers in shaders, which allowed me to finish this one up, for the most part (at least to the point where I don't have to keep poking away at it). It's a real beast of a shader and doesn't run fullspeed on the Intel HD4000 I've been testing with, but it looks pretty good in slow motion :P

You can download the shader and a couple of LUTs here. (same as the link at the top of the post)

I included 240p and 480p LUTs with both horizontal and vertical phosphor orientations. The 240phoriz LUT will give you essentially the image I was going for at the start of this post:

Notice, these shots look a bit dark, due to the way the LUT gets shrunken on my 1080p/4x scale screen. In the shader, there's a line you can uncomment when you're on a low-resolution screen like this to make the colors a bit brighter, which gives you something like this:

For anyone on a larger screen (or if you'd like to play around with the higher-resolution LUTs), I also included 480p variants, the vertical version of which looks like this:

Again, these shots would be brighter with the low-res mode enabled.

Last, just for fun, I made some screenshots of Super Metroid using the low-res option with the 240phoriz LUT, just like the Photoshop renders I made above, and here's how it looks:

Not bad! :D

One thing that still needs work is that the gaussian blur is only happening vertically, I think, but I'm not really sure why. I'm assuming I didn't put the horizontal pass into the framebuffer properly, so I'll keep fiddling with that. Also, certain emulation cores (such as snes9x-next) don't really play nicely with it, presumably due to horizontal resolution issues. Genesis Plus GX seems to work fine, at least:

 

Update (3/29/2013): I made a couple of small changes to the shader and a few different variations. I added in a new variable called 'brightness' (toward the end of the file) that you can raise/lower to increase/decrease the brightness of the image if it's too dark (within a range). Here's a pastebin of the updated default shader (shown with 240pvert LUT):

And here's one with scanlines added (sry, no screenshot; I'll try to add one soon):

And here's a super-mega version with Themaister's multipass NTSC shader+scanlines (another missing screenshot).

GPDP made some new LUTs, too, which is awesome. The first one mimics an aperture grill (click to download the LUT; shown here with scanlines added):

This one will look best at 9x scale instead of 10x.

The second LUT is actually pretty good at smaller scales, such as 4x, and takes the same idea as cgwg's CRT shader in that it tints alternating pixels green or magenta (click to download the LUT;

 I also modified cgwg's CRT shader to use the phosphor LUTs instead of the built-in pink/green pixel tint (note: screen transformation doesn't work right in this version) . It looks like this (click to download; shown with 240phoriz LUT):

Just as a warning: there's no reason to use this one instead of the regular CRT shader at less-than-huge scales. It looks significantly worse than the normal version. TBH, that's true for all of the shaders in this post...

And here's one with aliaspider's GTU shader replacing the gaussian passes (click to download; shown with 480pvert LUT):

Update (5/9/2013): Here's how the updated Cg port looks. I now consider this the best version and recommend users upgrade to it if they've been using the older GLSL version:

These shots were all taken at 4x, which would never have been possible with the GLSL version (without it looking like total garbage). It also handles pseudo-hires transparency decently well (see: Kirby shot), though you'll notice there's quite a bit of color mangling still...

You can download this shader here.

Fix for TV Overscan in Ubuntu with Nvidia Drivers

Ok, this problem has been a consistent thorn in my side for as long as I've been running Ubuntu on my TV (a number of years). On a lot of HDTVs, setting your computer to display 1080p or 720p or whatever will often cut off the edges of the screen. This cutoff is known as overscan and it used to be necessary because all sorts of garbage data would be displayed in the edges of broadcast signals and video games. Nowadays, that doesn't happen and we want to see every bit of the picture but many TVs still do it.

The easiest fix on TVs of any reasonable quality is to set the TV to "just scan" or "pixel perfect" mode (it goes by many different names depending on manufacturer). However, if your TV is cheap/crappy, you won't have such an option and the only way to deal with it is via software. This works just dandy in Windows, where the Nvidia driver has a handy little "overscan compensation" option where you can adjust a slider until your entire desktop is visible.

Sometimes, depending on driver revision and apparently some other unknown (to me) factors, a similar slider is available in the nvidia-settings applications in Ubuntu. This is undependable, though, and it will sometimes be grayed out or even disappear after you've used it in the past! :O

If you run into any of this, the way to fix it is to add a line to your /etc/X11/xorg.conf files that invokes the ViewPortIn/ViewPortOut options in the driver. So, open a terminal and type:

sudo nano /etc/X11/xorg.conf

You should see:

Section "Device"        Identifier    "Default Device"        Option        "NoLogo"       "True"EndSection

Or something similar. If you don't see any of that (e.g., if your file is totally empty), then just add it in manually.

Next, add a new line under the "NoLogo" line that looks like this (all one line, btw):

Option    "metamodes" "DFP-0: idealxresolution { ViewPortIn=idealxresolution,ViewPortOut=desiredxresolution+excessX+excessY }"

So, if, like me, you want it to run in 720p with 24 pixels of horizontal overscan compensation added to left/right sides and 14 pixels of vertical overscan compensation added to top/bottom, your line will look like this:

Option "metamodes" "DFP-1: 1280x720 { ViewPortIn=1280x720,ViewPortOut=1232x692+24+14 }"

Now, you'll notice that the buffers I've added to the end are exactly half of the total amount of resolution I'm chopping off. That's because the specified amount is added to both sides.

You'll also notice that my line uses DFP-1 instead of DFP-0. I'm not sure why that is, but that's what it identifies as in my nvidia-settings application, so be sure to check there first to find out what yours should be labeled as.

Log out and back in and it should use the new resolution. If not, try rebooting.

Fix for Win8 Unhandled Error -1 (SSF4)

Ugh, this problem was infuriating. I got a cheap copy of Windows 8 through my job and decided to go ahead and install it, just to be up-to-date. Then, I went to play SSF4AE via Steam and instead of launching, I got an ugly little window that said "An unhandled error has occurred -1."

After digging around online, it appears that the fault lies in Windows 8's default implementation of Games For Windows Live. |:-[

To get around this problem, we need to uninstall the default GFWL and download the latest installer from Microsoft.

You DO NOT have to uninstall/reinstall Steam as is suggested elsewhere online. If you try that, be aware that it will may uninstall all of your games at the same time (thanks to MasterPhW for the correction). It's supposed to ask before doing so, but it didn't ask me, so YMMV.

Background Shader for RetroArch

UPDATE (9/18/2012): It turns out the Themaister already wrote a waaay better version of this shader that can do some cool things like move the location of the game window and scale to arbitrary scale factors (i.e., not just even integers). I also made another old TV background:

If you'd like to try Themaister's version of the shader--which I've provided as-is, as well as combined with flat and curved CRT shaders--you can download it and the above TV background here:
http://www.mediafire.com/?av2y788pn3v1b3e

UPDATE (9/26/2012): Another one by request, using Super Metroid's Japanese artwork:

Download:

http://www.mediafire.com/?ah2a1hxptte886l

Original Post:

I've been working on a fun little pixel shader for use with RetroArch that functions similarly to the Cg border shader but using GLSL and the XML shader format. The way it works is that it combines the functionality of the existing integer scale shader with a lookup texture (LUT) that loads an external texture and then combines that image with the rendered game image. The LUT then has a transparent window in the middle where you can see the game peeking through.

In the simplest application, you can put whatever fancy stuff you want around the game window, like this awesome pixel-art border created by FirebrandX (NOTE: all of these look better in actual use; the way I took the screenshots caused the game's vertical resolution to get cut off by the border a bit):

By using RetroArch's awesome multipass shader support, you can also tack on some fun effects, like cgwg's CRT shader and some high-resolution artwork:

We can also do fun stuff like this TV border:

And we can use the transparency of the background image to overlay things across the game, like this Super Mario Bros 2 border, which has the characters overlapping a bit:

You can download these and others here:

http://www.mediafire.com/?1bss7814eupav1b

To use the shader, you'll need to uncheck the 'Lock Aspect Ratio' option in RetroArch, under 'Video' and put the desired background image in the same folder as the shader with filename 'background.png.' The backgrounds only work with specific monitor resolutions, so I've included 1080p versions of the ones pictured above, while FirebrandX was kind enough to make several iterations of his sweet pixel-art backgrounds that work at their specified resolutions. I've also included several background-compatible shaders, including the raw shader, ones with cgwg's CRT+NTSC-RGB (both flat and curved varieties) and some with cgwg's CRT+NTSC-RF (both flat and curved varieties).

In case you'd like to make your own, I've also included a simple template (in PSD format) for 1080p resolutions.

Ubuntu Quantal Beta - First Impressions on x120e

I just updated to the latest beta release of Ubuntu, codenamed Quantal Quetzel (or, as I call it, Quantum Pretzel), on my Lenovo x120e netbook.

Installation went smoothly and my optional broadcom wireless chipset was detected and enabled automatically during installation. This is the first time that has loaded without any hassle on my part, and I'm very pleased with the result. Users with the default atheros chipset have been enjoying this ease of installation for several releases already.

Installation went smoothly, now that they've squashed a particularly nasty bug in the installer that would crash if you tried to manually modify the partition table...

Once I booted into the actual installed system, it defaulted to use the open source driver for my integrated Radeon Fusion chipset, which provides adequate acceleration for transparency and other desktop composition goodies. Speaking of, the default Unity interface seems to have been improved further beyond the already much-improved Precise release, with the icons in the dock being smaller and less cartoony on my machine, with transparent backgrounds rather than the garish multicolored backgrounds from before (YMMV).

Virtual desktop switching also seems much improved. I never used it before because it felt clunky and laggy, but now transitions are smooth and dragging windows among desktops is effortless and intuitive. However, sometimes the windowing system gets confused and tries to maximize things inappropriately, leaving a transparent orange overlay on the screen--to show where it's trying to maximize--until you click on the window decoration and let it do its thing, then resize.

Suspend works well on this machine, faster than before and with quicker wakeups, and the wireless reconnects faster after suspend, as well. There doesn't seem to be a hibernate option anymore, so I couldn't test that.

One problem I ran into: I previously used my /etc/fstab to mount a shared folder from my network at startup via smbfs, but that package has been removed upstream, so now I use cifs as the filesystem and that seems to work just fine.

If you have any questions or anything you'd like me to test, feel free to hit me up in the comments.

Netflix Channel Installation Error on Softmodded Wii

The Netflix channel on my Wii finally stopped letting me skip the nag screen and demanded that I update. So, I deleted the old Netflix channel from Wii's the built-in channel management menu and attempted to download the new version from the Shop Channel.

I was greeted with an error 204036 and given the option to try again, which gave the same results each time.

So, I tried to just download a copy of the channel wad (Netflix Channel v1024 USA) to install through a wad manager. This didn't work, either, as it kept giving me error -1022 (Content did not match hash in TMD).

I tried using a bunch of different IOSes to install, but nothing worked, though I ended up finding a solution:

Step 1. Go download a copy of the old Netflix Channel wad (NetflixChannel-v1-USA) and reinstall it via a wad manager.

Step 2. Once it's installed successfully, use the wad manager to uninstall it. (I got an error during uninstallation, but it finished okay anyway, FYI)

Step 3. Install the new Netflix Channel wad and it should complete successfully this time.

Apparently, the Wii's normal channel removal tool leaves behind some garbage that stops the new channel from installing properly. By reinstalling the old version and then uninstalling it through the wad manager, we can get rid of that garbage and end up with a clean slate.

Update: If this doesn't work for you right away, you may have to install IOS 56 and/or the Shop Channel v20 first. In my case, I'm still on a much older shop channel, but someone in the comments mentioned needing to do this before it would work.

Update2: It's asking for another update... If you follow the steps above, then uninstall Netflix via a wad manager, you should be able to download the updated Netflix channel from the Shop Channel.