On application servers, gems are usually bundled into the project directory, at a location shared across deployments.

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

• Prevents you from data breaches and liability risks
• Upgrade at your own pace
• Works with modern Rubies

Read more Show archive.org snapshot

This is usually shared/bundle inside your project's root directory, e.g. /var/www/your-project/shared/bundle/.
If you can't find that, take a look at current/.bundle/config and look for BUNDLE_PATH.

When you are changing the version of RubyGems or Bundler on a system where gems are installed this way, you must wipe that bundle directory in addition to the user and system gems or gems that are already installed will not be reinstalled for the new RubyGems or Bundler.

Some gems may include binaries that are turned into binstubs by RubyGems and thus implicitly rely on a certain RubyGems version. Especially when downgrading Rubygems, you will encounter issues.

Note that removing the directory will prevent new application workers from booting up. You need to deploy soon after.

Also see: Fix Rubygems binary error: undefined method `activate_bin_path' for Gem:Module (NoMethodError)