That's about it...
I think Pilgrim dislikes the miners because Pilgrim considers himself cultured and educated, and likewise considers the miners uncultured and ignorant.
That kind of personal issues will eat a person up, I tell you.
I kinda get the intimation that it goes beyond that.
The character of Pilgrim represents the sort of person whose sense of self-identification (and self-valuation) depends preponderantly upon his vision of himself as intrinsically superior. It's the essence of the "aristocratic" impulse.
For such a critter to feel contentment, he needs
hoi polloi to denigrate and despise. Ain't no way to see the glitter if there ain't no mud in the picture against which to get a contrast, right? The only place for him to find that on Vesta is among the miners and secondary services providers who support the mining economy, the
nyekulturny grubbers-after-goodies in the floating islands hitherabouts.
What you're claiming
"will eat a person up" is for Pilgrim and his ilk the very breath of life itself.
Now, if you're looking for that which eats Pilgrim and the Mascons the proverbial new asshole, consider that the competent, capable, wealth-generating miners look upon them as ridiculous idiots, and simply don't give a damn about 'em. The Mascons are pretty much completely irrelevant to anything the real Belters - Pilgrim's
hoi polloi - value or need.
That, children, applied to Pilgrim, "
will eat a person [of the Mascon type]
up."