In this video Matt Cutt’s discusses the quality of Google results when backlinks are not considered as a ranking factor. You can tell that Matt has a hard time answering this question. Pretty much looks like it pains him to even consider it. Matt has spent his entire life defending backlinks as a legitimate ranking factor, and it appears that some day he might lose that battle.
In this video, Matt goes on record saying that they have tested looking at Google results when backlinks are not a ranking factor. The result he said was search results that were very low quality.
A couple things to keep in mind… Everything is subjective. So low quality to one person might be high quality to another. Clearly, you cannot just remove links as a ranking factor, you need to fill that gap with something else.