For the quick-open button to stop showing on every page whenever you highlight a piece of text you should just:
Open your Collabwriting Extension
Press the ⚙️ icon on the top right
Scroll down to the Preferences section
Check the "Hide quick open button" setting item
After this Collabwriting highlighter button won't show anymore on web pages you visit.