This is, ultimately, a very personal thing. Not just your preferences, but how you use the binocular. And the only way to really know is to try for yourself, NOT just a quick side-by-side in the store but actual extended use in the field.
My personal assessment: For casual use, in good light, there's not that much difference (assuming you have an 8x32 that you find comfortable to use on its own merits). However, in poor/harsh lighting, and/or with extended use for hours on end in the field, and especially if I (or my eyes) am tired, I find the difference in pure viewing comfort to be pretty huge.
Also, it's like any tool or gadget hobby -- a dedicated enthusiast is going to notice tiny differences that a casual user won't. And someone who uses the tool for hours on end every day will notice differences that someone who only uses a few minutes a day will not. That's why I think an 8x32 is more than good enough for the vast majority of end users, who will neither use them for hours a day nor nitpick the tiny differences like the obsessive enthusiasts.
There's also the caveat that I even have the 7x42 to compare to.... for most of my birding life I would just have a single binocular (nearly always 8x42 or 8x32) and, in the absence of direct comparison, I wouldn't notice the "discomfort" I'm describing. If all I had was, say, a Swaro SV 8x32, I would probably never think that it had poor viewing comfort. I know plenty of terrific birders who use an 8x32 as their primary binocular and it doesn't hold them back, you adapt and get used to what you're using.
But, long story short, it's a pretty simple concept. You're trying to line up one circle (your own pupil) with another circle (the binocular exit pupil). The bigger the exit pupil, the less precise you have to be to. Like shooting a basketball through a hoop -- the bigger the hoop, the less you have to focus and be perfect on every shot. Over hours of use, putting your binoculars up to your eyes hundreds of times, the tiny bit of extra effort accumulates into more eye fatigue.