Michael Crouse 9cd7696cdf [TFLite] Add tokenizer support for TFLite. ...

This third_party additional enables the standard tokenization algorithms
for text-based ML models for TFLite.

The support library will enable the new language detection model and
multiple other features under design and implementation.

TFLite Chrome Browser Eng Review: https://groups.google.com/a/google.com/g/chrome-eng-review/c/jNnAeHXw8_0


Bug: 1171995
Change-Id: Ida60ca763f8d151b3e7dcb833b4108793e75c971
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2658595
Reviewed-by: Alex Gough <ajgo@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: John Abd-El-Malek <jam@chromium.org>
Reviewed-by: Sophie Chang <sophiechang@chromium.org>
Commit-Queue: Michael Crouse <mcrouse@chromium.org>
Cr-Commit-Position: refs/heads/master@{#851273}

2021-02-05 21:01:18 +00:00

..

keras

…

metrics

…

_api_cache.json

…

_toc.yaml

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

all_symbols.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

BertTokenizer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

case_fold_utf8.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

coerce_to_structurally_valid_utf8.md

…

combine_segments.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

Detokenizer.md

…

find_source_offsets.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

FirstNItemSelector.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

gather_with_default.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

greedy_constrained_sequence.md

…

HubModuleSplitter.md

…

HubModuleTokenizer.md

…

keras.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

mask_language_model.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

MaskValuesChooser.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

max_spanning_tree_gradient.md

…

max_spanning_tree.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

metrics.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

ngrams.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

normalize_utf8_with_offsets_map.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

normalize_utf8.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

pad_along_dimension.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

pad_model_inputs.md

…

RandomItemSelector.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

Reduction.md

…

regex_split_with_offsets.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

regex_split.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

RegexSplitter.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

sentence_fragments.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

SentencepieceTokenizer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

sliding_window.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

span_alignment.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

span_overlaps.md

…

SplitMergeFromLogitsTokenizer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

SplitMergeTokenizer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

Splitter.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

StateBasedSentenceBreaker.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

Tokenizer.md

…

TokenizerWithOffsets.md

…

UnicodeCharTokenizer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

UnicodeScriptTokenizer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

viterbi_constrained_sequence.md

…

WaterfallTrimmer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

WhitespaceTokenizer.md

…

WordpieceTokenizer.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

WordShape_cls.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00

wordshape.md

[TFLite] Add tokenizer support for TFLite.

2021-02-05 21:01:18 +00:00