Doug wrote an article about the future of Free software gaming that is going to be sort of a jumping off place for this one. Remember that free software is not about price. Free software is about the freedom that the user retains when obtaining, installing, and using the software.
I agree that open source engines, while always a good thing, are not going to be what moves FLOSS gaming into the mainstream. A good engine is likely to result in better gameplay, but only if game creators utilize all of the engine’s capabilities to implement better concepts and designs.
I want to look at it from more a business standpoint, because I don’t play many games. In fact, I bought myself a PlayStation 3 in 2008, but it immediately disappeared into the youngest’s bedroom where it sat next to his XBox 360. He and the XBox are out of state, attending college, but my PlayStation is still in his old room. [Update: We're a PS4 family now. The Xbox 360 and PS3 are gone. I have a PS4. The oldest and the youngest each have PS4s.]
Core to the business of gaming is the idea that if you don’t lock the system and software down, the customers will copy and distribute the product (games) and the developers will go out of business. There is plenty of precedent for this. In the 1980s, I lived near a facility that handled products for Mattel’s game business. At the time, Mattel’s Intellivision was a distant second place to Atari’s 2600 VCS, but Intellivision had some really interesting games. There was a plant employee who would come to a certain store about twice per month and offer stolen game cartridges in the parking lot for $5 each. Eventually, Mattel ported many of its games to Atari’s platform, selling them under the “M Network” name, and this employee started offering them as well.
Maybe I shouldn’t admit it, but I think I bought about 60 cartridges from him. I think I had about 35 cartridges that I paid full price for. (I had both game systems.) This means that 2/3 of my game cartridges were not legitimately purchased.
One of the selling points when the PS3 came out was that you could install Linux on it and dual-boot with the original OS. Later, Sony released an update which removed that capability. They claimed that having an open, configurable system made the PS3 insecure. After you remove the technobabble and PR-speak, they were afraid that game software companies would abandon the PS3 unless Sony could guarantee that users could not use Linux to copy, alter, and distribute their wares.
I said all this to say that X11 is far from perfect, but X11 with the newest OpenGL could produce very acceptable graphics, if graphics card vendors like Nvidia and AMD/ATI would produce equally-functional drivers for Linux-based (or even BSD) operating systems. X11 is not the reason there have not been many quality games for Linux in the past, and I do not believe that the introduction of Wayland or Mir to the general Linux-using population will substantially change the quality or the number of games available.
Steam began producing games for some GNU+Linux based operating systems (primarily Ubuntu, I believe) in the past year or two. They did this despite Ubuntu’s still-existing dependence upon X11, simply because they view GNU+Linux as the only viable future for PC gaming. The Windows 8 experience is quite different from, and I would say quite inferior to, the traditional desktop operating system experience. As a consequence of this (as well as the popularity of mobile devices such as tablets), PC sales are sharply down. I’m not afraid to state my observation that Windows 8 is killing the PC market upon which Steam depends.
[As an aside, I upgraded my Windows 8 laptop to Windows 8.1. If you're on a laptop or desktop system without a touchscreen, Windows 8 is not very usable. Windows 8.1 offers little improvement. It is almost as though Microsoft is afraid to let people experience a non-touchscreen-centric Windows. I can see why Steam is trying to establish their brand on non-Windows systems, because Microsoft is heading towards being the company whose software is on "the work computer" but not anywhere else.]
From the standpoint of a business, TUR (technological usage restrictions), often euphemized as DRM (digital rights management or digital restrictions management), is often believed to be essential to preventing unpaide and unlicensed distribution of games. This, rather than any specific issues with X11 or desktop GNU+Linux distributions, accounts for the lack of available games.
Steam may be implementing its own TUR system. Whether they do implement TUR or choose not to do so, Steam’s success will prove to other PC games companies that GNU+Linux is a viable platform for their products. (Yes, I hope the GNU+Linux port of their proprietary game system successfully gains widespread adoption. And yes, I hope that is achieved without relying on technological restrictions.)
Game companies will have to figure out ways to utilize unpaid distribution and gamers who did not pay for the games to make their products ubiquitous. When “Game A” is so widely distributed and used that its availability becomes a selling point for the platform, the company that produces that game will benefit, even if not every player is a customer.
From a business standpoint, a successful Steam without TUR will create a rather large opening for free software gaming. I do not think a free software engine, in the absence of a breakthrough game that proves that TUR-free gaming is profitable on GNU+Linux, is enough to make free software gaming popular. I do not think that spectacular graphics will be enough, either. The key is to get game companies producing both FLOSS and proprietary games for the platform.
Now, free software games may also exist on other platforms. But on Windows they may have to overcome the negative experience of many freeware and shareware games in order to become popular. Also, just as on GNU+Linux, the key is still to prove to game developers that TUR-free applications can still be profitable.