BAZOMBO: a discursive miscellany of topics

Sanitising smelly text

Tuesday, December 04 2007

At work we are migrating an old site to a new CMS.

Unfortunately the content is a mess. Owing to people pasting text in from Word and various other accidents, one fragment of HTML can be a mixture of UTF-8 and Latin-1 and cp1252 and goodness knows what else. When you’ve been a good boy and coded all your templates to declare “I am UTF-8, honest guv” it’s a bit trying. Especially when the client complains.

The markup is pretty broken too. It’s littered with weird markup from Word and generally non-compliant.

So far I’m having good results from a pipeline of various tricks.

1. Python’s unicode function. unicode takes a string and transcodes it into Unicode. You can optionally force it to treat input as a particular encoding, and you can tell it how to handle errors.
2. Beautiful Soup. It finds tag soup delicious. It also does a best-effort to detect encodings and transcode to Unicode. (You have to love software with a module called UnicodeDammit).
3. htmltidy, in its utidylib manifestation. Does beautiful cleanup. It’s not super-robust though; I can make it segfault and dump core by feeding it the crap we have. Which is why I clean up with BeautifulSoup first.
4. I butchered Josh Goldfoot’s marvellous XSS-defense script to strip out some of the more outrageous markup that I know we won’t use.

The only downside is that over thousands of items, this is pretty slow. But it’s the price you pay to be beautiful, I guess.

no comments

Tags: python ~ unicode ~ markup ~ programming ~ html tidy ~ beautiful soup

HTMLTidy for Python

Friday, September 28 2007

When importing all my old content, I hit a snag. A lot of vital.org.nz has pretty broken markup in it. Burble’s templating system is strict XML under the hood, so any post or comment that contains broken markup causes burble to barf.

I discovered that there is a lovely Python wrapper for HTML Tidy. And there’s even an Ubuntu package. Problem solved.

>>> import tidy
>>> html = 'some <b>horrible<i> soup</b> which is nasty & yukky'
>>> options = {'show-body-only':'y', 'output-xhtml':'y', 'enclose-block-text':'y', 'enclose-text':'y'}
>>> body = str(tidy.parseString(html, **options))
>>> body
'<p>some <b>horrible <i>soup</i> which is nasty &amp; yukky</b></p>\n\n'


no comments

Tags: burble ~ python ~ html tidy ~ xml

---