public class DictionaryCompoundWordTokenFilter extends CompoundWordTokenFilterBase
TokenFilter that decomposes compound words found in many Germanic languages.
"Donaudampfschiff" becomes Donau, dampf, schiff so that you can find "Donaudampfschiff" even when you only enter "schiff". It uses a brute-force algorithm to achieve this.
You must specify the required Version compatibility when creating
CompoundWordTokenFilterBase:
If you pass in a CharArraySet as dictionary,
it should be case-insensitive unless it contains only lowercased entries and you
have LowerCaseFilter before this filter in your analysis chain.
For optional performance (as this filter does lots of lookups to the dictionary,
you should use the latter analysis chain/CharArraySet). Be aware: If you supply arbitrary
Sets to the ctors or String[] dictionaries, they will be automatically
transformed to case-insensitive!
CompoundWordTokenFilterBase.CompoundTokenDEFAULT_MAX_SUBWORD_SIZE, DEFAULT_MIN_SUBWORD_SIZE, DEFAULT_MIN_WORD_SIZE, dictionary, maxSubwordSize, minSubwordSize, minWordSize, offsetAtt, onlyLongestMatch, termAtt, tokens| Constructor and Description |
|---|
DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input,
Set dictionary)
Deprecated.
|
DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input,
Set dictionary,
int minWordSize,
int minSubwordSize,
int maxSubwordSize,
boolean onlyLongestMatch)
Deprecated.
|
DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input,
String[] dictionary)
Deprecated.
|
DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input,
String[] dictionary,
int minWordSize,
int minSubwordSize,
int maxSubwordSize,
boolean onlyLongestMatch)
Deprecated.
|
DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion,
org.apache.lucene.analysis.TokenStream input,
Set<?> dictionary)
Creates a new
DictionaryCompoundWordTokenFilter |
DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion,
org.apache.lucene.analysis.TokenStream input,
Set<?> dictionary,
int minWordSize,
int minSubwordSize,
int maxSubwordSize,
boolean onlyLongestMatch)
Creates a new
DictionaryCompoundWordTokenFilter |
DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion,
org.apache.lucene.analysis.TokenStream input,
String[] dictionary)
Deprecated.
Use the constructors taking
Set |
DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion,
org.apache.lucene.analysis.TokenStream input,
String[] dictionary,
int minWordSize,
int minSubwordSize,
int maxSubwordSize,
boolean onlyLongestMatch)
Deprecated.
Use the constructors taking
Set |
| Modifier and Type | Method and Description |
|---|---|
protected void |
decompose()
Decomposes the current
CompoundWordTokenFilterBase.termAtt and places CompoundWordTokenFilterBase.CompoundToken instances in the CompoundWordTokenFilterBase.tokens list. |
incrementToken, makeDictionary, resetaddAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString@Deprecated public DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
DictionaryCompoundWordTokenFilter(Version, TokenStream, String[], int, int, int, boolean) insteadDictionaryCompoundWordTokenFilter.input - the TokenStream to processdictionary - the word dictionary to match againstminWordSize - only words longer than this get processedminSubwordSize - only subwords longer than this get to the output streammaxSubwordSize - only subwords shorter than this get to the output streamonlyLongestMatch - Add only the longest matching subword to the stream@Deprecated public DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input, String[] dictionary)
DictionaryCompoundWordTokenFilter(Version, TokenStream, String[]) insteadDictionaryCompoundWordTokenFilterinput - the TokenStream to processdictionary - the word dictionary to match against@Deprecated public DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input, Set dictionary)
DictionaryCompoundWordTokenFilter(Version, TokenStream, Set) insteadDictionaryCompoundWordTokenFilterinput - the TokenStream to processdictionary - the word dictionary to match against. If this is a CharArraySet it must have set ignoreCase=false and only contain
lower case strings.@Deprecated public DictionaryCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream input, Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
DictionaryCompoundWordTokenFilter(Version, TokenStream, Set, int, int, int, boolean) insteadDictionaryCompoundWordTokenFilterinput - the TokenStream to processdictionary - the word dictionary to match against. If this is a CharArraySet it must have set ignoreCase=false and only contain
lower case strings.minWordSize - only words longer than this get processedminSubwordSize - only subwords longer than this get to the output streammaxSubwordSize - only subwords shorter than this get to the output streamonlyLongestMatch - Add only the longest matching subword to the stream@Deprecated public DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
SetDictionaryCompoundWordTokenFiltermatchVersion - Lucene version to enable correct Unicode 4.0 behavior in the
dictionaries if Version > 3.0. See CompoundWordTokenFilterBase for details.input - the TokenStream to processdictionary - the word dictionary to match againstminWordSize - only words longer than this get processedminSubwordSize - only subwords longer than this get to the output streammaxSubwordSize - only subwords shorter than this get to the output streamonlyLongestMatch - Add only the longest matching subword to the stream@Deprecated public DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, String[] dictionary)
SetDictionaryCompoundWordTokenFiltermatchVersion - Lucene version to enable correct Unicode 4.0 behavior in the
dictionaries if Version > 3.0. See CompoundWordTokenFilterBase for details.input - the TokenStream to processdictionary - the word dictionary to match againstpublic DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion,
org.apache.lucene.analysis.TokenStream input,
Set<?> dictionary)
DictionaryCompoundWordTokenFiltermatchVersion - Lucene version to enable correct Unicode 4.0 behavior in the
dictionaries if Version > 3.0. See CompoundWordTokenFilterBase for details.input - the TokenStream to processdictionary - the word dictionary to match against.public DictionaryCompoundWordTokenFilter(org.apache.lucene.util.Version matchVersion,
org.apache.lucene.analysis.TokenStream input,
Set<?> dictionary,
int minWordSize,
int minSubwordSize,
int maxSubwordSize,
boolean onlyLongestMatch)
DictionaryCompoundWordTokenFiltermatchVersion - Lucene version to enable correct Unicode 4.0 behavior in the
dictionaries if Version > 3.0. See CompoundWordTokenFilterBase for details.input - the TokenStream to processdictionary - the word dictionary to match against.minWordSize - only words longer than this get processedminSubwordSize - only subwords longer than this get to the output streammaxSubwordSize - only subwords shorter than this get to the output streamonlyLongestMatch - Add only the longest matching subword to the streamprotected void decompose()
CompoundWordTokenFilterBaseCompoundWordTokenFilterBase.termAtt and places CompoundWordTokenFilterBase.CompoundToken instances in the CompoundWordTokenFilterBase.tokens list.
The original token may not be placed in the list, as it is automatically passed through this filter.decompose in class CompoundWordTokenFilterBaseCopyright © 2000-2015 Apache Software Foundation. All Rights Reserved.