strutils - Text manipulation

So much practical programming involves string manipulation, which Python readily accommodates. Still, there are dozens of basic and common capabilities missing from the standard library, several of them provided by strutils.

boltons.strutils.camel2under(camel_string)[source]

Converts a camelcased string to underscores. Useful for turning a class name into a function name.

>>> camel2under('BasicParseTest')
'basic_parse_test'
boltons.strutils.under2camel(under_string)[source]

Converts an underscored string to camelcased. Useful for turning a function name into a class name.

>>> under2camel('complex_tokenizer')
'ComplexTokenizer'
boltons.strutils.slugify(text, delim='_', lower=True, ascii=False)[source]

A basic function that turns text full of scary characters (i.e., punctuation and whitespace), into a relatively safe lowercased string separated only by the delimiter specified by delim, which defaults to _.

The ascii convenience flag will asciify() the slug if you require ascii-only slugs.

>>> slugify('First post! Hi!!!!~1    ')
'first_post_hi_1'
>>> slugify("Kurt Gödel's pretty cool.", ascii=True) ==         b'kurt_goedel_s_pretty_cool'
True
boltons.strutils.split_punct_ws(text)[source]

While str.split() will split on whitespace, split_punct_ws() will split on punctuation and whitespace. This used internally by slugify(), above.

>>> split_punct_ws('First post! Hi!!!!~1    ')
['First', 'post', 'Hi', '1']
boltons.strutils.unit_len(sized_iterable, unit_noun='item')[source]

Returns a plain-English description of an iterable’s len(), conditionally pluralized with cardinalize(), detailed below.

>>> print(unit_len(range(10), 'number'))
10 numbers
>>> print(unit_len('aeiou', 'vowel'))
5 vowels
>>> print(unit_len([], 'worry'))
No worries
boltons.strutils.ordinalize(number, ext_only=False)[source]

Turns number into its cardinal form, i.e., 1st, 2nd, 3rd, 4th, etc. If the last character isn’t a digit, it returns the string value unchanged.

Parameters:
  • number (int or str) – Number to be cardinalized.
  • ext_only (bool) – Whether to return only the suffix. Default False.
>>> print(ordinalize(1))
1st
>>> print(ordinalize(3694839230))
3694839230th
>>> print(ordinalize('hi'))
hi
>>> print(ordinalize(1515))
1515th
boltons.strutils.cardinalize(unit_noun, count)[source]

Conditionally pluralizes a singular word unit_noun if count is not one, preserving case when possible.

>>> vowels = 'aeiou'
>>> print(len(vowels), cardinalize('vowel', len(vowels)))
5 vowels
>>> print(3, cardinalize('Wish', 3))
3 Wishes
boltons.strutils.pluralize(word)[source]

Semi-intelligently converts an English word from singular form to plural, preserving case pattern.

>>> pluralize('friend')
'friends'
>>> pluralize('enemy')
'enemies'
>>> pluralize('Sheep')
'Sheep'
boltons.strutils.singularize(word)[source]

Semi-intelligently converts an English plural word to its singular form, preserving case pattern.

>>> singularize('chances')
'chance'
>>> singularize('Activities')
'Activity'
>>> singularize('Glasses')
'Glass'
>>> singularize('FEET')
'FOOT'
boltons.strutils.asciify(text, ignore=False)[source]

Converts a unicode or bytestring, text, into a bytestring with just ascii characters. Performs basic deaccenting for all you Europhiles out there.

Also, a gentle reminder that this is a utility, primarily meant for slugification. Whenever possible, make your application work with unicode, not against it.

Parameters:
  • text (str or unicode) – The string to be asciified.
  • ignore (bool) – Configures final encoding to ignore remaining unasciified unicode instead of replacing it.
>>> asciify('Beyoncé') == b'Beyonce'
True
boltons.strutils.is_ascii(text)[source]

Check if a unicode or bytestring, text, is composed of ascii characters only. Raises ValueError if argument is not text.

Parameters:text (str or unicode) – The string to be checked.
>>> is_ascii('Beyoncé')
False
>>> is_ascii('Beyonce')
True
boltons.strutils.is_uuid(obj, version=4)[source]

Check the argument is either a valid UUID object or string.

Parameters:
  • obj (object) – The test target. Strings and UUID objects supported.
  • version (int) – The target UUID version, set to 0 to skip version check.
>>> is_uuid('e682ccca-5a4c-4ef2-9711-73f9ad1e15ea')
True
>>> is_uuid('0221f0d9-d4b9-11e5-a478-10ddb1c2feb9')
False
>>> is_uuid('0221f0d9-d4b9-11e5-a478-10ddb1c2feb9', version=1)
True
boltons.strutils.html2text(html)[source]

Strips tags from HTML text, returning markup-free text. Also, does a best effort replacement of entities like “ ”

>>> r = html2text(u'<a href="#">Test &amp;<em>(\u0394&#x03b7;&#956;&#x03CE;)</em></a>')
>>> r == u'Test &(\u0394\u03b7\u03bc\u03ce)'
True
boltons.strutils.strip_ansi(text)[source]

Strips ANSI escape codes from text. Useful for the occasional time when a log or redirected output accidentally captures console color codes and the like.

>>> strip_ansi('art')
'art'

Supports unicode, str, bytes and bytearray content as input. Returns the same type as the input.

There’s a lot of ANSI art available for testing on sixteencolors.net. This function does not interpret or render ANSI art, but you can do so with ansi2img or escapes.js.

boltons.strutils.bytes2human(nbytes, ndigits=0)[source]

Turns an integer value of nbytes into a human readable format. Set ndigits to control how many digits after the decimal point should be shown (default 0).

>>> bytes2human(128991)
'126K'
>>> bytes2human(100001221)
'95M'
>>> bytes2human(0, 2)
'0.00B'
boltons.strutils.find_hashtags(string)[source]

Finds and returns all hashtags in a string, with the hashmark removed. Supports full-width hashmarks for Asian languages and does not false-positive on URL anchors.

>>> find_hashtags('#atag http://asite/#ananchor')
['atag']

find_hashtags also works with unicode hashtags.

boltons.strutils.a10n(string)[source]

That thing where “internationalization” becomes “i18n”, what’s it called? Abbreviation? Oh wait, no: a10n. (It’s actually a form of numeronym.)

>>> a10n('abbreviation')
'a10n'
>>> a10n('internationalization')
'i18n'
>>> a10n('')
''
boltons.strutils.gzip_bytes(bytestring, level=6)[source]

Turn some bytes into some compressed bytes.

>>> len(gzip_bytes(b'a' * 10000))
46
Parameters:
  • bytestring (bytes) – Bytes to be compressed
  • level (int) – An integer, 1-9, controlling the speed/compression. 1 is fastest, least compressed, 9 is slowest, but most compressed.

Note that all levels of gzip are pretty fast these days, though it’s not really a competitor in compression, at any level.

boltons.strutils.gunzip_bytes(bytestring)[source]

The gzip module is great if you have a file or file-like object, but what if you just have bytes. StringIO is one possibility, but it’s often faster, easier, and simpler to just use this one-liner. Use this tried-and-true utility function to decompress gzip from bytes.

>>> gunzip_bytes(_EMPTY_GZIP_BYTES) == b''
True
>>> gunzip_bytes(_NON_EMPTY_GZIP_BYTES).rstrip() == b'bytesahoy!'
True
boltons.strutils.iter_splitlines(text)[source]

Like str.splitlines(), but returns an iterator of lines instead of a list. Also similar to file.next(), as that also lazily reads and yields lines from a file.

This function works with a variety of line endings, but as always, be careful when mixing line endings within a file.

>>> list(iter_splitlines('\nhi\nbye\n'))
['', 'hi', 'bye', '']
>>> list(iter_splitlines('\r\nhi\rbye\r\n'))
['', 'hi', 'bye', '']
>>> list(iter_splitlines(''))
[]
boltons.strutils.indent(text, margin, newline='\n', key=<type 'bool'>)[source]

The missing counterpart to the built-in textwrap.dedent().

Parameters:
  • text (str) – The text to indent.
  • margin (str) – The string to prepend to each line.
  • newline (str) – The newline used to rejoin the lines (default: \n)
  • key (callable) – Called on each line to determine whether to indent it. Default: bool, to ensure that empty lines do not get whitespace added.
boltons.strutils.escape_shell_args(args, sep=' ', style=None)[source]

Returns an escaped version of each string in args, according to style.

Parameters:
  • args (list) – A list of arguments to escape and join together
  • sep (str) – The separator used to join the escaped arguments.
  • style (str) – The style of escaping to use. Can be one of cmd or sh, geared toward Windows and Linux/BSD/etc., respectively. If style is None, then it is picked according to the system platform.

See args2cmd() and args2sh() for details and example output for each style.

boltons.strutils.args2cmd(args, sep=' ')[source]

Return a shell-escaped string version of args, separated by sep, using the same rules as the Microsoft C runtime.

>>> print(args2cmd(['aa', '[bb]', "cc'cc", 'dd"dd']))
aa [bb] cc'cc dd\"dd

As you can see, escaping is through backslashing and not quoting, and double quotes are the only special character. See the comment in the code for more details. Based on internal code from the subprocess module.

boltons.strutils.args2sh(args, sep=' ')[source]

Return a shell-escaped string version of args, separated by sep, based on the rules of sh, bash, and other shells in the Linux/BSD/MacOS ecosystem.

>>> print(args2sh(['aa', '[bb]', "cc'cc", 'dd"dd']))
aa '[bb]' 'cc'"'"'cc' 'dd"dd'

As you can see, arguments with no special characters are not escaped, arguments with special characters are quoted with single quotes, and single quotes themselves are quoted with double quotes. Double quotes are handled like any other special character.

Based on code from the pipes/shlex modules. Also note that shlex and argparse have functions to split and parse strings escaped in this manner.

boltons.strutils.parse_int_list(range_string, delim=', ', range_delim='-')[source]

Returns a sorted list of positive integers based on range_string. Reverse of format_int_list().

Parameters:
  • range_string (str) – String of comma separated positive integers or ranges (e.g. ‘1,2,4-6,8’). Typical of a custom page range string used in printer dialogs.
  • delim (char) – Defaults to ‘,’. Separates integers and contiguous ranges of integers.
  • range_delim (char) – Defaults to ‘-’. Indicates a contiguous range of integers.
>>> parse_int_list('1,3,5-8,10-11,15')
[1, 3, 5, 6, 7, 8, 10, 11, 15]
boltons.strutils.format_int_list(int_list, delim=', ', range_delim='-', delim_space=False)[source]

Returns a sorted range string from a list of positive integers (int_list). Contiguous ranges of integers are collapsed to min and max values. Reverse of parse_int_list().

Parameters:
  • int_list (list) – List of positive integers to be converted into a range string (e.g. [1,2,4,5,6,8]).
  • delim (char) – Defaults to ‘,’. Separates integers and contiguous ranges of integers.
  • range_delim (char) – Defaults to ‘-’. Indicates a contiguous range of integers.
  • delim_space (bool) – Defaults to False. If True, adds a space after all delim characters.
>>> format_int_list([1,3,5,6,7,8,10,11,15])
'1,3,5-8,10-11,15'
class boltons.strutils.MultiReplace(sub_map, **kwargs)[source]

MultiReplace is a tool for doing multiple find/replace actions in one pass.

Given a mapping of values to be replaced it allows for all of the matching values to be replaced in a single pass which can save a lot of performance on very large strings. In addition to simple replace, it also allows for replacing based on regular expressions.

Keyword Arguments:

Parameters:
  • regex (bool) – Treat search keys as regular expressions [Default: False]
  • flags (int) – flags to pass to the regex engine during compile

Dictionary Usage:

from lrmslib import stringutils
s = stringutils.MultiReplace({
    'foo': 'zoo',
    'cat': 'hat',
    'bat': 'kraken'
})
new = s.sub('The foo bar cat ate a bat')
new == 'The zoo bar hat ate a kraken'

Iterable Usage:

from lrmslib import stringutils
s = stringutils.MultiReplace([
    ('foo', 'zoo'),
    ('cat', 'hat'),
    ('bat', 'kraken)'
])
new = s.sub('The foo bar cat ate a bat')
new == 'The zoo bar hat ate a kraken'

The constructor can be passed a dictionary or other mapping as well as an iterable of tuples. If given an iterable, the substitution will be run in the order the replacement values are specified in the iterable. This is also true if it is given an OrderedDict. If given a dictionary then the order will be non-deterministic:

>>> 'foo bar baz'.replace('foo', 'baz').replace('baz', 'bar')
'bar bar bar'
>>> m = MultiReplace({'foo': 'baz', 'baz': 'bar'})
>>> m.sub('foo bar baz')
'baz bar bar'

This is because the order of replacement can matter if you’re inserting something that might be replaced by a later substitution. Pay attention and if you need to rely on order then consider using a list of tuples instead of a dictionary.

sub(text)[source]

Run substitutions on the input text.

Given an input string, run all substitutions given in the constructor.

boltons.strutils.multi_replace(text, sub_map, **kwargs)[source]

Shortcut function to invoke MultiReplace in a single call.

boltons.strutils.unwrap_text(text, ending='\n\n')[source]

Unwrap text, the natural complement to textwrap.wrap().

>>> text = "Short \n lines  \nwrapped\nsmall.\n\nAnother\nparagraph."
>>> unwrap_text(text)
'Short lines wrapped small.\n\nAnother paragraph.'
Parameters:
  • text – A string to unwrap.
  • ending (str) – The string to join all unwrapped paragraphs by. Pass None to get the list. Defaults to ‘nn’ for compatibility with Markdown and RST.