Is there something wrong if one does not follow any of the 4 imams? Instead one follows only the Quran and Sunnah.

Wrong, because you need to reach the status of ijtihad to do that. That is you need to recall all text revealed, the principles laid out by sahabas and also ijma` and qiyaas all at one time so as to be followed. I have been personally involving myself in research for more than 15 years now, but still I feel more of following the one who was in the best of eras. The holy prophet (saw) said: the best of era is mine, then the next generation, then the next generation. Then lies would spread. As such follow an imam is more reassuringly safe that we are following the right path.