Using Vim to repair files with incorrect character encoding/representation

Posted Over 12 years ago. Visible to the public.

Consider you have a file that uses improper encoding on special characters. Example: You see the latin1 version "ñ" for the UTF-8 "ñ" but the file itself is stored as UTF-8 (meaning that the UTF-8 bytes are doubly encoded).

You can fix that easily with Vim:

vim broken.file

Now you tell vim that the file's encoding is actually latin1 (you can see what Vim is currently using by saying only :set fileencoding):

:set fileencoding=latin1

Write and reload the file:

:w
:e

All should be good now.

Adjust to your needs for any other encodings.

Arne Hartherz
makandra.com
Say thanks3
Last edit
About 12 years ago
License
Source code in this card is licensed under the MIT License.

Related cards:

Customize tokenization of the MySQL FULLTEXT parser

The way MySQL's FULLTEXT tokenizer splits text into word tokens might not always be what you need. E.g. it splits a word at period characters.

Since the tokenizer has near-zero configuration options (minimum word length and stopwords list), you n...

How to make Rational#to_s return strings without denominator 1 again

The way Rational#to_s works on Ruby has changed from Ruby 1.9 on. Here is how to get the old behavior back.

You may want this for things where Rationals are being used, like when subtracting Date objects from one another.

What's happening?...

Vim: How to write a file you opened without sudo

Have you ever opened a file with vim, edited it and when you wanted to save your changes it told you "Can't open file for writing", because you opened it without sudo? There's an easy hack to let you write it anyway:

:w !sudo tee %

The...

OpenAI TTS: How to generate audio samples with more than 4096 characters

OpenAI is currently limiting the Audio generating API endpoint to text bodies with a maximum of 4096 characters.
You can work around that limit by splitting the text into smaller fragments and stitch together the resulting mp3 files with a CLI too...

See with tig which git commits touch a file or files or folders 

tig path_to_file_or_files_or_path_with_wildcard

Tod: A Gem for handling daytime without a date

Tod is a gem for working with daytimes.

Another additional gem?

Thus SQL has a time datatype for storing time of day in the format hh:mm:ss, neither Ruby nor Rails themselves offer an elegant way to deal with day times.

Time and `Dat...

How to cycle through grep results with vim

grep is the go-to CLI tool to accomplish tasks like filtering large files for arbitrary keywords. When additional context is needed for search results, you might find yourself adding flags like -B5 -A10 to your query. Now, every search result ...

How to not leave trailing whitespace (using your editor or Git)

There is no reason to leave trailing whitespace characters in your project's files, so don't add any.

A git diff --check will tell you if there are any and you should not commit when you see them. So go ahead and switch your editor/IDE to autom...

How to fix: Unable to read files from a VirtualBox shared host folder

When you want to copy/move from a shared folder (on Windows guests, for example) and it fails with absurd error messages (lots of text or an error about the target being read-only), you are probably running on Guest Additions that no longer match ...

Dynamically uploading files to Rails with jQuery File Upload

Say we want …

  • to create a Gallery that has a name and has_many :images, which in turn have a caption
  • to offer the user a single form to create a gallery with any number of images
  • immediate uploads with a progress bar per image
  • a snapp...
Posted by Arne Hartherz to makandra dev (2012-02-06 13:43)
This website uses short-lived cookies to improve usability.
or learn more