Exactly, you don’t need a JavaScript framework all the time.

First, if you only render HTML from the server, why do you need a JS framework? Just serve the full page and cache it.

Secondly, unlike with JSON data structure, HTML is pretty hard to traverse.

And you can’t really build a rich UI from snippets of HTML

Lastly, if you have more than one client responding with HTML is wasteful and probably unnecessary.

I really can’t imagine a scenario where you need to return a button element from an API

I’d say this is actually an example of premature optimization. Solving a performance problem, which doesn’t exist.