Package org.apache.lucene.analysis.ngram
Class EdgeNGramTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.ngram.EdgeNGramTokenizer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public final class EdgeNGramTokenizer extends org.apache.lucene.analysis.Tokenizer
Tokenizes the input from an edge into n-grams of given size(s).This
Tokenizer
create n-grams from the beginning edge or ending edge of a input token. MaxGram can't be larger than 1024 because of limitation.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
EdgeNGramTokenizer.Side
Specifies which side of the input the n-gram should be generated from
-
Field Summary
Fields Modifier and Type Field Description static int
DEFAULT_MAX_GRAM_SIZE
static int
DEFAULT_MIN_GRAM_SIZE
static EdgeNGramTokenizer.Side
DEFAULT_SIDE
-
Constructor Summary
Constructors Constructor Description EdgeNGramTokenizer(Reader input, String sideLabel, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given rangeEdgeNGramTokenizer(Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given rangeEdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, Reader input, String sideLabel, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given rangeEdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given rangeEdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source, Reader input, String sideLabel, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given rangeEdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source, Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
end()
boolean
incrementToken()
Returns the next token in the stream, or null at EOS.void
reset()
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
-
-
-
-
Field Detail
-
DEFAULT_SIDE
public static final EdgeNGramTokenizer.Side DEFAULT_SIDE
-
DEFAULT_MAX_GRAM_SIZE
public static final int DEFAULT_MAX_GRAM_SIZE
- See Also:
- Constant Field Values
-
DEFAULT_MIN_GRAM_SIZE
public static final int DEFAULT_MIN_GRAM_SIZE
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
EdgeNGramTokenizer
public EdgeNGramTokenizer(Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range- Parameters:
input
-Reader
holding the input to be tokenizedside
- theEdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
-
EdgeNGramTokenizer
public EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source, Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range- Parameters:
source
-AttributeSource
to useinput
-Reader
holding the input to be tokenizedside
- theEdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
-
EdgeNGramTokenizer
public EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, Reader input, EdgeNGramTokenizer.Side side, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range- Parameters:
factory
-AttributeSource.AttributeFactory
to useinput
-Reader
holding the input to be tokenizedside
- theEdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
-
EdgeNGramTokenizer
public EdgeNGramTokenizer(Reader input, String sideLabel, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range- Parameters:
input
-Reader
holding the input to be tokenizedsideLabel
- the name of theEdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
-
EdgeNGramTokenizer
public EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource source, Reader input, String sideLabel, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range- Parameters:
source
-AttributeSource
to useinput
-Reader
holding the input to be tokenizedsideLabel
- the name of theEdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
-
EdgeNGramTokenizer
public EdgeNGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, Reader input, String sideLabel, int minGram, int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range- Parameters:
factory
-AttributeSource.AttributeFactory
to useinput
-Reader
holding the input to be tokenizedsideLabel
- the name of theEdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
-
-
Method Detail
-
incrementToken
public boolean incrementToken() throws IOException
Returns the next token in the stream, or null at EOS.- Specified by:
incrementToken
in classorg.apache.lucene.analysis.TokenStream
- Throws:
IOException
-
end
public void end()
- Overrides:
end
in classorg.apache.lucene.analysis.TokenStream
-
reset
public void reset() throws IOException
- Overrides:
reset
in classorg.apache.lucene.analysis.TokenStream
- Throws:
IOException
-
-