Which of these two phrases is the correct one to use when you want to say "I want to be happy", is either of them grammatically incorrect and what's the difference between them?

Mi volas estu feliĉa
Mi volas esti feliĉa

"Mi volas estu feliĉa" is not grammatical.
"Mi volas esti feliĉa" is more or less the same as "mi volas, ke mi estu feliĉa".

"Esti" is the infinitive, while "estu" is the optative and imperative.

In your example, you can use the infinitive only, when the person wishing is the same person being happy. Otherwise you need a subordinate clause.

Mi volas esti feliĉa = Mi volas, ke mi estu feliĉa.
Mi volas, ke vi estu feliĉa. (Not possible with infinitive)

Let us take a different sentence in English first to illustrate what a modal verb is. In the sentence "He must do that", the verb "must" is such, in English the other verb must be in infinitive, you cannot say "He must does that". Esperanto works the same way, you must say Li devas fari tion instead of Li devas faras tion.

The verb volas also works the same way, so you say Mi volas esti feliĉa, I want to be happy. In English you can command yourself or express a wish, "I want me to be happy" which is equivalent to "I want that I am happy" except in English the first way is the preferred way. This latter cannot be expressed in Esperanto by Mi volas, ke mi estas feliĉa, because the as-ending shows a real action, something that actually happens, but here we have a wish. To express that wish Esperanto uses something called volative, volitivo. With that volative you can say Mi volas, ke mi estu feliĉa, I want me to be happy.

See the grammar for the volative here in Lernu.