One of the main problems with existing open source speech recognition systems is that they are not really designed to be used in end-user software. They are mainly research projects created by universities and they are intended to support new research. They allow to quickly add with new features and get best results for various evaluations.
The end-user software doesn't work like that, you might not need to demonstrate the best accuracy but you need to match the user expectations. For example, user expects to get a reasonable result even if he speaks far from the microphone or whispers the words. No modern system can recognize whisper reliably, thus mismatched expectation, thus complains. A lot of work is required to solve all the problems like this.
However, since many commercial companies promoted speech recognition to end-users, open source software also got a chance. We can build software for mass-market and match commercial solutions in terms of accuracy and robustness. Important step here would be to gain the audience attention, instead of software for geeks we need to become a software for everybody, a very hard problem to solve. It's great there are projects with big ambitions here, in particular Mozilla Foundation.
Recently Mozilla Foundation announced a project to support WebSpeech HTML API in their browsers. Celebrating 10 years of Firefox development, Mozilla CTO, Andreas Gal, announced this and many other features coming in Mozilla codebase. During Google Summer of Code project by Andre Natal a base system was implemented and Andre continues work on the project. You can get some ideas of where it is now and how it developers from the following post. So we will probably be able to see speech interface in Firefox browser and Firefox OS pretty soon.
One of the main issue in wide adoption of the speech interfaces would be the support for big and small languages. Firefox considers this as important direction of development, in particular support for Indian languages. I hope we are going to see a lot of progress here.