I must have missed clearing cache earlier after applying the patch, the patch...

I must have missed clearing cache earlier after applying the patch, the patch is working fine and ensures that module doesn't cause any problems when using with admin toolbar. Hiding the links shouldn't be a problem as developers can provide logout link anywhere and ensure users are able to logout easily.

I am not sure if event subscriber suggested in #21 would be a better solution, but the patch provided in this issue does solve our problem.