History of tokenize @13b58c20-1d59-4f6e-9490-ff339014f685/main
- Add fuzzing to the `tokenize` package Joseph Kato 4 years ago
- Clean up tests Joseph Kato 6 years ago
- Update link to TreebankWordTokenizer source Joseph Kato (commit: GitHub) 6 years ago
- Remove TODOs from GoDoc Joseph Kato 6 years ago
- Drop matloob/regexp Joseph Kato 6 years ago
- Only call `FindAllStringSubmatchIndex` once Joseph Kato 6 years ago
- More caching in abbreviationReplacer Joseph Kato 6 years ago
- Cache compiled regexps in abbreviationReplacer Joseph Kato 6 years ago
- Use matloob.io/regexp Joseph Kato 6 years ago
- Update comments Joseph Kato 6 years ago
- Lowercase `rule` Joseph Kato 6 years ago
- Leave `rule.sub` early for non-matches Joseph Kato 6 years ago
- Leave `search` early if we can Joseph Kato 6 years ago
- Precompile boundaries Joseph Kato 6 years ago
- Add benchmarks Joseph Kato 6 years ago
- Update comments Joseph Kato 6 years ago
- Add the PragmaticSegmenter Joseph Kato 6 years ago
- Use modified PunktSentenceTokenizer in TextToWords Joseph Kato 6 years ago
- Add TestNewRegexpTokenizer Joseph Kato 6 years ago
- More tests Joseph Kato 6 years ago
- Add TestTextToWords Joseph Kato 6 years ago
- Fix example tests Joseph Kato 6 years ago
- Move doc examples to tokenize_test Joseph Kato 6 years ago
- Clean up Godoc Joseph Kato 6 years ago
- Update licenses Joseph Kato 6 years ago
- Remove "-" from WordBoundaryTokenizer Joseph Kato 6 years ago
- Improve PunktSentenceTokenizer Joseph Kato 6 years ago
- tag: add license to aptag Joseph Kato 7 years ago
- summarize: Update `TreebankWordTokenizer` citation Joseph Kato 7 years ago
- tokenize: Describe `TreebankWordTokenizer` better Joseph Kato 7 years ago