It seems to me that you have not allowed for the fact that the pencils are consumable items and the next ones might be a slightly different size. The suggestion by Jeff, jhiker using flaps in bending inside the holes seems a much better solution. In my experience, your high friction arrangement of a parallel plain hole in polyamide and a round painted pencil will have no tolerance between too tight and too loose, particularly as polyamides swell in high humidity. Having small sections such as 4 flaps or many narrow splines means that you can build a tight interference fit and have to deform only a few square millimetres of plastic rather than the 3 square inches of contact between each pencil and the round hole holder. Much more consistent and wider tolerances.
Sebastian's suggestion of tapering the hole is similar, reducing the contact area will allow for a tighter interference fit and wider tolerances but you should check the manufacturing tolerance of the pencils first. It could be outside the limits needed for a reasonable fit in a tapered hole. Of course hexagonal pencils will allow greater tolerance in the hole diameter as they have very little contact area with the hole. A hexagonal pencil in a tapered hole could work well.
As a manufacturing engineer, I would see the O ring as a good toolroom solution, rather than an acceptable production solution with extra parts and extra assembly operations.
Jim