What if you have obligations along with the entrepreneurial bug? Trust me, it can be very frustrating.
A lot of people wonder if there is a way out. A lot of people recommend to follow your passion with all your energies. One way out is to work and build on your idea until it demands all your time (Redbus is a classic example). Note that perseverance becomes your strength.
If that is the only option, so be it. One way or the other, just do it.