They overestimated the value of the hostages IMO
They thought it would be a Gilad Shalit situation x100 and that Netanyahu would fold but it's now obvious they didn't expect him to go respond hard. The disregard Bibi has had for its own hostages with how much he bombed Gaza was clearly not expected.
I think the fact most casualties/hostages of Oct 7th were civilians played a huge part in how hard Netanyahu reacted. I'm of the opinion that if they had solely taken/killed military/reservists, things would have played way differently.
I think the current reporting is that Hamas expected much more resistance from the Israeli forces once Operation Al Aqsa Flood had started, and the situation on the ground broke out into disorder with many non-Hamas individuals/forces joining the fray leading to breakdown of order. But I don't think went into the operation with the goal of a large scale hostage exchange based on Israeli largesse, they were trying to disrupt the burgeoning regional consensus of normalization with the Israeli state. In this way, Netanyahu going so hard bolstered their case and cause.
As for the Israeli response, I think the relatively high level of civilian casualties/hostages only gave Israel the opening to do what they've always sought to do; eradicate the Palestinians. I agree that it might have played out differently if Hamas had more operational control over the Palestinian forces engaged on October 7th and the result was a limiting of the targets to the Israeli military, but only because that would have robbed Israel of the cover of bloodlust vegenace that convinced the US and Western nations to allow them to move ahead with the genocide. This is why they mainstream Western media was complicit in this genocide; they guilelessly laundered the most extreme Israeli claims (babies in ovens, mass systemic rape, etc).