@Deprecated public final class CJKTokenizer extends Tokenizer
The tokens returned are every two adjacent characters with overlap match.
Example: "java C1C2C3C4" will be segmented to: "java" "C1C2" "C2C3" "C3C4".
Additionally, the following is applied to Latin text (such as English):AttributeSource.AttributeFactory, AttributeSource.State| Constructor and Description |
|---|
CJKTokenizer(AttributeSource.AttributeFactory factory,
Reader in)
Deprecated.
|
CJKTokenizer(AttributeSource source,
Reader in)
Deprecated.
|
CJKTokenizer(Reader in)
Deprecated.
Construct a token stream processing the given input.
|
| Modifier and Type | Method and Description |
|---|---|
void |
end()
Deprecated.
This method is called by the consumer after the last token has been
consumed, after
TokenStream.incrementToken() returned false
(using the new TokenStream API). |
boolean |
incrementToken()
Deprecated.
Returns true for the next token in the stream, or false at EOS.
|
void |
reset()
Deprecated.
Resets this stream to the beginning.
|
void |
reset(Reader reader)
Deprecated.
Expert: Reset the tokenizer to a new reader.
|
close, correctOffsetaddAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toStringpublic CJKTokenizer(Reader in)
in - I/O readerpublic CJKTokenizer(AttributeSource source, Reader in)
public CJKTokenizer(AttributeSource.AttributeFactory factory, Reader in)
public boolean incrementToken()
throws IOException
incrementToken in class TokenStreamIOException - - throw IOException when read error public final void end()
TokenStreamTokenStream.incrementToken() returned false
(using the new TokenStream API). Streams implementing the old API
should upgrade to use this feature.
This method can be used to perform any end-of-stream operations, such as
setting the final offset of a stream. The final offset of a stream might
differ from the offset of the last token eg in case one or more whitespaces
followed after the last token, but a WhitespaceTokenizer was used.end in class TokenStreampublic void reset()
throws IOException
TokenStreamTokenStream.reset() is not needed for
the standard indexing process. However, if the tokens of a
TokenStream are intended to be consumed more than once, it is
necessary to implement TokenStream.reset(). Note that if your TokenStream
caches tokens and feeds them back again after a reset, it is imperative
that you clone the tokens when you store them away (on the first pass) as
well as when you return them (on future passes after TokenStream.reset()).reset in class TokenStreamIOExceptionpublic void reset(Reader reader) throws IOException
Tokenizerreset in class TokenizerIOExceptionCopyright © 2000-2015 Apache Software Foundation. All Rights Reserved.