We tried to invade during the American Revolution. It was largely a fiasco, and the citizenry was openly hostile towards the new American government.
We gave it another go during the War of 1812. The American military leadership was completely incompetent, the Canadians were still hostile, and we got our asses handed to us. By the time new, more competent leadership manifested itself e.g. Winfield Scott, Alexander Macomb, we were on the defensive.
IMO, the last opportunity for a serious invasion of Canada was during the US Civil War, where Confederate Agents and various malcontents were using it as a staging ground. This led the UK to massively increase their defensive posture in Canada i.e. forts and additional forces.
Note that we kept war plans for the invasion of Canada ready--just in case, you know.