At work we are migrating an old site to a new CMS.

Unfortunately the content is a mess. Owing to people pasting text in from Word and various other accidents, one fragment of HTML can be a mixture of UTF-8 and Latin-1 and cp1252 and goodness knows what else. When you’ve been a good boy and coded all your templates to declare “I am UTF-8, honest guv” it’s a bit trying. Especially when the client complains.

The markup is pretty broken too. It’s littered with weird markup from Word and generally non-compliant.

So far I’m having good results from a pipeline of various tricks.

1. Python’s unicode function. unicode takes a string and transcodes it into Unicode. You can optionally force it to treat input as a particular encoding, and you can tell it how to handle errors.
2. Beautiful Soup. It finds tag soup delicious. It also does a best-effort to detect encodings and transcode to Unicode. (You have to love software with a module called UnicodeDammit).
3. htmltidy, in its utidylib manifestation. Does beautiful cleanup. It’s not super-robust though; I can make it segfault and dump core by feeding it the crap we have. Which is why I clean up with BeautifulSoup first.
4. I butchered Josh Goldfoot’s marvellous XSS-defense script to strip out some of the more outrageous markup that I know we won’t use.

The only downside is that over thousands of items, this is pretty slow. But it’s the price you pay to be beautiful, I guess.