Awesome
Strs
Strs.jl is a container for a number of different packages from JuliaString It has two main goals:
- To be a drop-in replacement for the built-in
String
andChar
types, adding types that are both faster and easier to use, that are also using for interfacing with other languages, and are safer to use. - To have a better option than the built-in string literal syntax and the
@printf
/@sprintf
macros for formatted output.
It brings together the following:
-
A better type of string literal, using StrLiterals, StrFormat, and StrEntities
This is of the form
f"..."
orF"..."
.This uses Swift-style
\
escape sequences, such as\u{xxxx}
for Unicode constants, instead of\uXXXX
and\UXXXXXXXX
, which have the advantage of not having to worry about some digit or letter A-F or a-f occurring after the last hex digit of the Unicode constant.It also means that
$
, a very common character for LaTeX strings or output of currencies, does not need to be in a string quoted as '$'It uses
\(expr)
for interpolation like Swift, instead of$name
or$(expr)
, which also has the advantage of not having to worry about the next character in the string someday being allowed in a name.It allows for embedding Unicode characters using a variety of easy to remember names, instead of hex codes:
\:emojiname:
\<latexname>
\N{unicodename}
\&htmlname;
Examples of this are:f"\<dagger> \¥ \N{ACCOUNT OF} \:snake:"
, which returns the string:"† ¥ ℀ 🐍 "
Default formatting based on the type of the argument: i.e.
f"\(1.23)"
returns"1.23"
, butf"\%(1.23)"
returns"1.230000"
based on the default format set up for AbstractFloat types. See Format for more information on how to set up defaults for your own types, or to change the defaults for floats, strings, etc.It also supports C-style formatting, but without having to count the position of the argument: i.e. instead of
@sprintf("$name has \$%10.8f", 1.23)"
,f"\(name) has $\%10.8f(1.23)"
-
Faster and more flexible set of string and character types, such as
ASCIIStr
,Latin1Str
,UCS2Str
,UTF32Str
, that are indexed by character, which can be much easier to use than to have deal withnextind
andprevind
, etc. This alleviates a common source of bugs in Julia, where people are unfamiliar with the difference between indexing by the byte or codeunit offset, instead of by the codepoints (characters). Using these types in your code can help speed things up. -
Faster and VALIDATED
UTF8Str
type. Julia's built-inString
type allows storing invalid sequences. (Strs provides aText1Str
type for holding strings that might be invalid UTF-8, or might be some other encoding, such as Microsoft's CP1252) This is especially a problem because the built-in Regex support in Julia incorrectly passes a flag saying that the argument has already been checked and is a valid UTF-8 sequence. Skipping that check in PCRE2 does make regex matching much faster, however it leaves you open to attacks if your are using regex on unchecked string input. -
Types for Binary strings, as well as strings that are known to be text strings, but whose encoding is not known (might be UTF-8 with certain commonly accepted but invalid sequence, such as representing characters > uFFFF in 6 bytes, as two 16 bit surrogate characters, or encoding a null byte as
\0xc0\0x80
instead of\0
, or S-JIS, CP1252, etc.) -
Highly optimized string functions, operating on 2, 4, or 8 characters at a time (I do intend to optimize these further, by using vector instructions on Intel, ARM, and POWER architectures, to process up to 64 characters at a time).
-
Thread-safe Regex support (it was not thread-safe in the LTS (long term support) version of Julia, currently v1.05, but that has been fixed as of the v1.3 release)
-
Regex support that doesn't assume that
String
values are valid UTF-8, so that it can't be used as a way of attacking programs written in Julia by passing certain unvalidated strings to the PCRE2 library. For speed, one can use theUTF8Str
type instead ofString
usingR"..."
instead of ther"..."
.
I would very much appreciate any constructive criticism, help implementing some of the ideas, ideas on how to make it perform better, bikeshedding on names and API, etc. Also, I'd love contributions of benchmark code and/or samples for different use cases of strings, or pointers to such (such as a way to get lots of tweets, to test mixed text and emojis, for example).
Package | Release | Release Date | Package Evaluator | Unit Tests | Description |
---|---|---|---|---|---|
ModuleInterfaceTools | Tools to create a common API for all of these packages | ||||
StrAPI | Common API for string/character functionality | ||||
CharSetEncodings | Basic types/support for Character Sets, Encodings, and Character Set Encodings | ||||
ChrBase | Chr{CharSet,CodeUnitType} type and support | ||||
MurmurHash3 | Pure Julia implementation of MurmurHash3 | ||||
PCRE2 | PCRE2 library support | ||||
Format | Python/C style formatting (based on Formatting) | ||||
StrBase | Str{CSE, Hash, SubSet, Cache} type | ||||
StrRegex | Regex support for all string types | ||||
StrLiterals | Extensible string literal support | ||||
StrFormat | Formatting extensions for literals | ||||
StrTables | Low-level support for entity tables | ||||
HTML_Entities | HTML character sequences | ||||
Emoji_Entities | Emoji names (including composite ones) | ||||
LaTeX_Entities | Julia LaTeX character names | ||||
Unicode_Entities | Unicode standard character names | ||||
StrEntities | Entity extensions for literals | ||||
InternedStrings | Save space by interning strings (by @oxinabox!) |
The package ModuleInterfaceTools is used to set up a consistent and easy to use API for most of the cooperating packages, without having to worry too much about imports, exports, using, and what functions are part of a public API, and which ones are part of the internal development API for other packages to extend.
Architecture and Operations
The general philosophy of the architecture is as follows: have a single easy to use type that can replace String
that conforms to the recommendations of the Unicode Organization (which internally uses 4 types and is implemented currently as a Julia Union, and has O(1) indexing to characters, not just code units), as well as types to represent binary strings, raw unvalidated strings (made up of 1, 2, or 4 byte codepoints), as well as types for validated ASCII, Latin1, UCS2 (16-bit, BMP [Basic Multilingual Plane]), UTF-8, UTF-16, and UTF-32 encoded strings.
Optimizations for multi code unit encodings such as UTF-8 & UTF-16 will be moved to StrUTF8
and StrUTF16
packages (before splitting them out, I'll make sure that the functionality still works, only with slower generic methods, so that you only take up the extra space if you need the faster speed).
Extensions such as converting to and from non-Unicode encodings, such as Windows CP-1252 or China's official character set, GB18030, will be done in another package, StrEncodings
.
Subtypes that directly support things like substrings, caching hash values, and caching one or more versions of the string (such as the originally unmodified byte, 16-bit or 32-bit word stream, in the case where the input was not valid, or a valid UTF-8 (similar to the way Python can cache a UTF-8 version of a string) and/or UTF-16 encoded version, for better performance when interoperating with other languages such as JavaScript, Swift, Java, or OS APIs like Windows that expect UTF-16).
Since things like AbstractChar
and CodeUnits
which I originally implemented for this package have now been added to Base
(in master), I have moved support for those to a file to provide them for v0.6.2.
There is a codepoints
iterator, which iterates over the unsigned integer codepoints in a string
(for strings with the CodeUnitSingle
trait, it is basically the same as the codeunits
iterator).
Also in the works is using the new ability to add properties, in order to allow for different "views" of a string, no matter how it is stored internally, for example a mystring.utf8
view, or a mystring.utf16
view (that can use the internal cached copy if available, as an optimization).
Types
Currently, there are the following types:
-
Str
, which is the general, parameterized string type, and -
Chr
, the general, parameterized character type. -
BinaryStr
for storing non-textual data as a sequence of bytes. -
ASCIIStr
an ASCII string, composed ofASCIIChr
1-byte codepoints -
LatinStr
a string using the Latin1 subset of Unicode, composed ofLatinChr
1-byte codepoints. -
UCS2Str
a string composed of characters (UCS2Chr
s) only in the Unicode BMP, stored as 2 byte code units (that each store a single codepoint) -
UTF32Str
a string with only valid Unicode characters, 0-0xd7ff, 0xe000-0x10ffff, stored as 4 byte code units. -
UTF8Str
a string with only valid Unicode characters, the same asUTF32Str
, however encoded using UTF-8, conforming to the Unicode Organization's standard, i.e. no long encodings, surrogates, or invalid bytes. -
UTF16Str
a string similar to `UTF8Str, encoded via UTF-16, also conforming to the Unicode standard, i.e. no out of order or isolated surrogates. -
Text1Str
a text string that may contain any sequence of bytes -
Text2Str
a text string that may contain any sequence of 16-bit words -
Text4Str
a text string that may contain any sequence of 32-bit words -
UniStr
a Union type, that can be one of the following 4 types,ASCIIStr
, and 3 internal types: -
_LatinStr
a byte string that must contain at least one character > 0x7f -
_UCS2Str
a word string that must contain at least one character > 0xff -
_UTF32Str
a 32-bit word string that must contain at least one character > 0xffff
The only real difference in handling LatinStr
and _LatinStr
, is that uppercasing the characters 'µ': (Unicode U+00b5 (category Ll: Letter, lowercase)
and 'ÿ': Unicode U+00ff (category Ll: Letter, lowercase)
produces the BMP characters 'Μ': Unicode U+039c (category Lu: Letter, uppercase)
and 'Ÿ': Unicode U+0178 (category Lu: Letter, uppercase)
respectively in _LatinStr
, because it is just an optimization for storing the full Unicode character set, not the ANSI/ISO 8859-1 character set that ise used for the first 256 code points of Unicode.
Those three internal types should never be used directly, as indicated by the leading _
in the name.
For all of the built-in Str
types, there is a corresponding built-in character set encoding, i.e. BinaryCSE
, LatinCSE
, UTF8CSE
, etc.
There are also a number of similar built-in character sets defined (*CharSet
), and character types (*Chr
).
The cse
function returns the character set encoding for a string type or a string.
charset
returns the character set, and encoding
returns the encoding.
For example, cse(UTF8Str)
returns UTF8CSE
, charset(UTF8Str)
returns CharSet{UTF32}
,
encoding(UTF8Str)
return Encoding{UTF8}()
Functions
There is a new API that I am working on for indexing and searching, (however there is a problem on v0.7 due to the deprecation for find
being overbroad, and causing a type of type piracy, deprecating methods of types not in Base):
-
find(First, ...)
replacesfindfirst
-
find(Last, ...)
replacesfindlast
-
find(Next, ...)
replacesfindnext
-
find(Prev, ...)
replacesfindprev
-
index(str, pos)
replacesthisind(str, pos)
-
index(Next, ...
replacesnextind(...)
-
index(Prev, ...
replacesprevind(...)
Also there are more readable function names that always separate words with _
, and avoid hard to understand abbreviations:
-
is*
->is_*
(forascii
,digit
,space
,numeric
,valid
,defined
,empty
,assigned
) -
iscntrl
->is_control
-
isgraph
->is_graphic
-
isprint
->is_printable
-
ispunct
->is_punctuation
-
isalpha
->is_letter
-
isalnum
->is_alphanumeric
-
isgraphemebreak
->is_grapheme_break
-
isgraphemebreak!
->is_grapheme_break!
-
occursin
->occurs_in
-
textwidth
->text_width
-
lowercasefirst
->lowercase_first
-
uppercasefirst
->uppercase_first
-
startswith
->starts_with
-
endswith
->ends_with
-
In addition, I've added
is_alphabetic
Kudos
Nobody is an island, and to achieve great things, one must stand on the shoulders of giants.
I would like to thank some of those giants in particular:
-
The four co-creators of Julia: Jeff Bezanson,Viral B. Shah, Alan Edelman, and Stefan Karpinski, without their uncompromising greediness, none of this would be possible.
-
Tom Breloff, for showing how an ecosystem could be created in Julia, i.e. "Build it, and they will come", for providing some nice code in this PR (which I shamelessly pirated in order to create Format, and for good advice at JuliaCon.
-
Ismael Venegas Castelló for encouraging me to tweet about Julia starting at the 2015 JuliaCon, for good advice, and being a great guy in general.
-
Chris Rackaukas simply a star in Julia now, great guy, great advice, and great blogs about stuff that's usually way over my head. Julia is incredibly lucky to have him.
-
Jacob Quinn, for collaborating & discussions early on in Strings on ideas for better string support in Julia, as well as a lot of hard work on things dear to me, such as databases and importing/exporting data SQLite, ODBC, CSV, WeakRefStrings, DataStreams, Feather, JSON2
-
Milan Bouchet-Valat, for discussions on string handling and encoding in StringEncodings
-
Tim Holy for the famous "Holy" Trait Trick, which I use extensively in the Str* packages, for the work along with Matt Bauman on making Julia arrays general, extensible while still performing well, and hence very useful in my work.
-
Steven G. Johnson for illuminating me on how one could create a whole package in very few lines of code when I first started learning Julia, see DecFP
-
Tony Kelman, for very thorough reviews of my PRs, I learned a great deal from his (and other Julians') comments), including that I didn't have to code in C anymore to get the performance I desired.
-
Lyndon White, I've already "appropriated" :grinning: his very nice InternedStrings into this package, I'm really lucky to have gotten him to join the organization!
-
Bogumił Kamiński who has been doing a great job testing and reviewing
Strs
(as well as doing the same for the string/character support in Julia Base), as well as input into the design. (Also very glad to have co-opted him to become a member of the org) -
Last but not least, Julia mathematical artist (and blogger!) extraordinaire, Cormullion, creator of our wonderful logo!
Also thanks to anybody who's answered my (sometimes stupid :grinning:) questions on Gitter and Discourse
Kudos to all of the other contributors to Julia and the ever expanding Julia ecosystem!