Sci-fi and fantasy serials are full of monsters of the week: introduced in one episode, forgotten the next. These creatures are, almost by their nature, unknown to most of the fictional society. No one has mentioned them before because the writers just thought of them! Just because Flash Gordon meets a tribe of rock-people in a cave doesn’t mean that the rock-people need to be fully integrated into galactic civilization.
Hobbits fit perfectly into the monster-of-the-week pattern. They live in an isolated area, and, outside the local area, no one has ever heard of them. It’s clear that, during the course of Gandalf’s episodic travels, he had a fun interlude in the Shire. In most high fantasy, it would have gotten a chapter at most. But here is Gandalf’s (and Tolkien’s) brilliance: Gandalf thought, “I think these guys have surprisingly good stats. I want to start making a big deal out of them.” And so he started sending them on missions of greater and greater importance. The Lord of the Rings saga is like a crazy version of Flash Gordon where the rock-people tribe take over the narrative for the rest of the series.But every D&D campaign doesn’t have to be Lord of the Rings.
Halflings are in a weird position in D&D: they’re one of the original four races (although Gygax didn’t like them very much). Therefore, they’re much more common in most D&D worlds than hobbits are in Middle Earth. They’ve grown in importance the same way Tatooine has in the Star Wars universe: instead of a backwater planet, it’s a ubiquitous stop on every video-game, novel, and prequel version of the galactic tour.
There’s a lot of D&D history behind us, so it’s probably fine at this point to accept halflings as one of the defining elements of the game. But it might be worth remembering that halflings don’t NEED to be one of the Big Four races in your world. Maybe there’s a middle ground between “50% of all thieves are halflings” and “halflings are banned from my game world.” On this middle ground (or middle earth, if you will) the big players would be humans, dwarves, and elves. If someone wants to be a halfling, or a dragonborn, or a drow, the DM will probably allow it, but they are all what 5e is calling “rare races.” They’re monsters of the week, allowed as occasional PCs, but not necessarily important to the setting. It’s OK if they’re there in the world somewhere… but they’re generally overlooked. I don’t know about halflings, but that’s how hobbits would prefer it.