edu.cmu.sphinx.linguist.language.ngram.large

Class BinaryLoader

java.lang.Object

edu.cmu.sphinx.linguist.language.ngram.large.BinaryLoader

Direct Known Subclasses:

BinaryStreamLoader

public class BinaryLoader
extends java.lang.Object

Reads a binary NGram language model file ("DMP file") generated by the SphinxBase sphinx_lm_convert.

Note that all probabilities in the grammar are stored in LogMath log base format. Language Probabilities in the language model file are stored in log 10 base. They are converted to the LogMath base.

Constructor Summary

Constructors

Constructor and Description
BinaryLoader(java.io.File location, java.lang.String format, boolean applyLanguageWeightAndWip, float languageWeight, double wip, float unigramWeight) Initializes the binary loader
BinaryLoader(java.lang.String format, boolean applyLanguageWeightAndWip, float languageWeight, double wip, float unigramWeight) Initializes the binary loader

Method Summary

Modifier and Type	Method and Description
void	deallocate()
boolean	getBigEndian() Returns true if the loaded file is in big-endian.
long	getBigramOffset() Returns the location (or offset) into the file where bigrams start.
float[]	getBigramProbabilities() Returns all the bigram probabilities.
int	getBytesPerField() Returns the multiplier for the size of a NGram (1 for 16 bits, 2 for 32 bits).
int	getLogBigramSegmentSize() Returns the log of the bigram segment size
int	getLogNGramSegmentSize() Returns the log of the NGram segment size
int	getMaxDepth() Returns the maximum depth of the language model
float[]	getNGramBackoffWeights(int n) Returns all the NGram backoff weights at a specified N order.
long	getNGramOffset(int n) Returns the location (or offset) into the file where NGrams start at a specified N order.
float[]	getNGramProbabilities(int n) Returns all the NGram probabilities at a specified N order.
int[]	getNGramSegments(int n) Returns the NGram segment table at a specified order.
int	getNumberBigrams() Returns the number of bigrams
int	getNumberNGrams(int n) Returns the number of NGrams at a specified N order.
int	getNumberTrigrams() Returns the number of trigrams
int	getNumberUnigrams() Returns the number of unigrams
float[]	getTrigramBackoffWeights() Returns all the trigram backoff weights
long	getTrigramOffset() Returns the location (or offset) into the file where trigrams start.
float[]	getTrigramProbabilities() Returns all the trigram probabilities.
int[]	getTrigramSegments() Returns the trigram segment table.
edu.cmu.sphinx.linguist.language.ngram.large.UnigramProbability[]	getUnigrams() Returns all the unigrams
java.lang.String[]	getWords() Returns all the words.
byte[]	loadBuffer(long position, int size) Loads the contents of the memory-mapped file starting at the given position and for the given size, into a byte buffer.
protected void	loadModelLayout(java.io.InputStream inputStream) Loads the language model from the given file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BinaryLoader

public BinaryLoader(java.io.File location,
                    java.lang.String format,
                    boolean applyLanguageWeightAndWip,
                    float languageWeight,
                    double wip,
                    float unigramWeight)
             throws java.io.IOException

Initializes the binary loader

Parameters:

location - location of the model

format - file format

applyLanguageWeightAndWip - if true apply language weight and word insertion penalty

languageWeight - language weight

wip - word insertion probability

unigramWeight - unigram weight

Throws:

java.io.IOException - if an I/O error occurs

BinaryLoader

public BinaryLoader(java.lang.String format,
                    boolean applyLanguageWeightAndWip,
                    float languageWeight,
                    double wip,
                    float unigramWeight)

Initializes the binary loader

Parameters:

format - file format

applyLanguageWeightAndWip - if true apply language weight and word insertion penalty

languageWeight - language weight

wip - word insertion probability

unigramWeight - unigram weight

Method Detail

deallocate

public void deallocate()
                throws java.io.IOException

Throws:

java.io.IOException

getNumberUnigrams

public int getNumberUnigrams()

Returns the number of unigrams

Returns:

the number of unigrams

getNumberBigrams

public int getNumberBigrams()

Returns the number of bigrams

Returns:

the number of bigrams

getNumberTrigrams

public int getNumberTrigrams()

Returns the number of trigrams

Returns:

the number of trigrams

getNumberNGrams

public int getNumberNGrams(int n)

Returns the number of NGrams at a specified N order.

Parameters:

n - the desired order

Returns:

the number of NGrams

getUnigrams

public edu.cmu.sphinx.linguist.language.ngram.large.UnigramProbability[] getUnigrams()

Returns all the unigrams

Returns:

all the unigrams

getBigramProbabilities

public float[] getBigramProbabilities()

Returns all the bigram probabilities.

Returns:

all the bigram probabilities

getTrigramProbabilities

public float[] getTrigramProbabilities()

Returns all the trigram probabilities.

Returns:

all the trigram probabilities

getTrigramBackoffWeights

public float[] getTrigramBackoffWeights()

Returns all the trigram backoff weights

Returns:

all the trigram backoff weights

getTrigramSegments

public int[] getTrigramSegments()

Returns the trigram segment table.

Returns:

the trigram segment table

getLogBigramSegmentSize

public int getLogBigramSegmentSize()

Returns the log of the bigram segment size

Returns:

the log of the bigram segment size

getNGramProbabilities

public float[] getNGramProbabilities(int n)

Returns all the NGram probabilities at a specified N order.

Parameters:

n - the desired order

Returns:

all the NGram probabilities

getNGramBackoffWeights

public float[] getNGramBackoffWeights(int n)

Returns all the NGram backoff weights at a specified N order.

Parameters:

n - the desired order

Returns:

all the NGram backoff weights

getNGramSegments

public int[] getNGramSegments(int n)

Returns the NGram segment table at a specified order.

Parameters:

n - the desired order

Returns:

the NGram segment table

getLogNGramSegmentSize

public int getLogNGramSegmentSize()

Returns the log of the NGram segment size

Returns:

the log of the NGram segment size

getWords

public java.lang.String[] getWords()

Returns all the words.

Returns:

all the words

getBigramOffset

public long getBigramOffset()

Returns the location (or offset) into the file where bigrams start.

Returns:

the location of the bigrams

getTrigramOffset

public long getTrigramOffset()

Returns the location (or offset) into the file where trigrams start.

Returns:

the location of the trigrams

getNGramOffset

public long getNGramOffset(int n)

Returns the location (or offset) into the file where NGrams start at a specified N order.

Parameters:

n - the desired order

Returns:

the location of the bigrams

getMaxDepth

public int getMaxDepth()

Returns the maximum depth of the language model

Returns:

the maximum depth of the language model

getBigEndian

public boolean getBigEndian()

Returns true if the loaded file is in big-endian.

Returns:

true if the loaded file is big-endian

getBytesPerField

public int getBytesPerField()

Returns the multiplier for the size of a NGram (1 for 16 bits, 2 for 32 bits).

Returns:

the multiplier for the size of a NGram

loadBuffer

public byte[] loadBuffer(long position,
                         int size)
                  throws java.io.IOException

Loads the contents of the memory-mapped file starting at the given position and for the given size, into a byte buffer. This method is implemented because MappedByteBuffer.load() does not work properly.

Parameters:

position - the starting position in the file

size - the number of bytes to load

Returns:

the loaded ByteBuffer

Throws:

java.io.IOException - if IO went wrong

loadModelLayout

protected void loadModelLayout(java.io.InputStream inputStream)
                        throws java.io.IOException

Loads the language model from the given file.

Parameters:

inputStream - stream to read the language model data

Throws:

java.io.IOException - if IO went wrong