Normally you can list all gems of the current ruby version with gem list, which also includes the gems of you Gemfile. These can be uninstalled with gem uninstall gemname.

List and uninstall a gem installed via Bundler from Github

This does not work for gems installed directly from Github. They do not appear in gem list.

Show all gems installed via Github by bundler:

ls ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/

Remove a gem installed via Github by Bundler:

rm -rf ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gemname-xyz
rm -rf ~/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/cache/bundler/gemname-1.2.3

This might also be true for any other gem installed via self defined Git source and not standard Rubygems.