re: Were the Japanese planning to invade Australia during WW2?

Yes, they would have invaded eventually, but because of the crippling defeats they suffered, it was impossible. To control the Pacific they would need to cut Australia off from the U.S. and GBR. While I don't think it would be impossible to completely control all of the continent, it would waste too many resources that could be allocated elsewhere.

In my opinion, Midway and Hawaii were more important than Australia. Cutting the US off from staging there would allow the IJN to have free reign and then, the possibility to invade Australia would open.