I guess it depends on the area you live and the cost to busy. I can see your perspective on houses in the 100-200's. Higher cost mean higher deposits. In these cases a person may not want to wait 3 to 5 years to save the deposit. Just giving an option if you have a healthy retirement plan
Then don't buy until you're ready. Taking out 401k loans to get a house (which also will be another loan to pay) is a sign you should wait until you can really afford it.
401k loans are an option. Imo Just a bad one for almost all circumstances. And also people tend to keep borrowing from their 401k, setting a bad habit.