WordFrequencyTransformer

Instance Constructors

new WordFrequencyTransformer(wordIndexBroadcast: Broadcast[Map[String, Int]], unigramCounts: Map[Int, Int])

wordIndexBroadcast
A mapping from token string to its frequency-ordered index
unigramCounts
the counts of unigrams in the training corpus

Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
final val OOV_INDEX: Int(-1)
final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: PipelineDataset[Seq[String]], labels: PipelineDataset[L]): Pipeline[Seq[String], C]

Chains a label estimator onto the end of this pipeline, producing a new pipeline.
Chains a label estimator onto the end of this pipeline, producing a new pipeline. If this pipeline has already been executed, it will not need to be fit again.
est
The estimator to chain onto the end of this pipeline
data
The training data to use (the estimator will be fit on the result of passing this data through the current pipeline)
labels
The labels to use when fitting the LabelEstimator. Must be zippable with the training data.

Definition Classes
Chainable
final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: RDD[Seq[String]], labels: PipelineDataset[L]): Pipeline[Seq[String], C]

Chains a label estimator onto the end of this pipeline, producing a new pipeline.
Chains a label estimator onto the end of this pipeline, producing a new pipeline. If this pipeline has already been executed, it will not need to be fit again.
est
The estimator to chain onto the end of this pipeline
data
The training data to use (the estimator will be fit on the result of passing this data through the current pipeline)
labels
The labels to use when fitting the LabelEstimator. Must be zippable with the training data.

Definition Classes
Chainable
final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: PipelineDataset[Seq[String]], labels: RDD[L]): Pipeline[Seq[String], C]

Chains a label estimator onto the end of this pipeline, producing a new pipeline.
Chains a label estimator onto the end of this pipeline, producing a new pipeline. If this pipeline has already been executed, it will not need to be fit again.
est
The estimator to chain onto the end of this pipeline
data
The training data to use (the estimator will be fit on the result of passing this data through the current pipeline)
labels
The labels to use when fitting the LabelEstimator. Must be zippable with the training data.

Definition Classes
Chainable
final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: RDD[Seq[String]], labels: RDD[L]): Pipeline[Seq[String], C]

Chains a label estimator onto the end of this pipeline, producing a new pipeline.
Chains a label estimator onto the end of this pipeline, producing a new pipeline. If this pipeline has already been executed, it will not need to be fit again.
est
The estimator to chain onto the end of this pipeline
data
The training data to use (the estimator will be fit on the result of passing this data through the current pipeline)
labels
The labels to use when fitting the LabelEstimator. Must be zippable with the training data.

Definition Classes
Chainable
final def andThen[C](est: Estimator[Seq[Int], C], data: PipelineDataset[Seq[String]]): Pipeline[Seq[String], C]

Chains an estimator onto the end of this pipeline, producing a new pipeline.
Chains an estimator onto the end of this pipeline, producing a new pipeline. If this pipeline has already been executed, it will not need to be fit again.
est
The estimator to chain onto the end of this pipeline
data
The training data to use (the estimator will be fit on the result of passing this data through the current pipeline)

Definition Classes
Chainable
final def andThen[C](est: Estimator[Seq[Int], C], data: RDD[Seq[String]]): Pipeline[Seq[String], C]

Chains an estimator onto the end of this pipeline, producing a new pipeline.
Chains an estimator onto the end of this pipeline, producing a new pipeline. If this pipeline has already been executed, it will not need to be fit again.
est
The estimator to chain onto the end of this pipeline
data
The training data to use (the estimator will be fit on the result of passing this data through the current pipeline)

Definition Classes
Chainable
final def andThen[C](next: Chainable[Seq[Int], C]): Pipeline[Seq[String], C]

Chains a pipeline onto the end of this one, producing a new pipeline.
Chains a pipeline onto the end of this one, producing a new pipeline. If either this pipeline or the following has already been executed, it will not need to be fit again.
next
the pipeline to chain

Definition Classes
Chainable
def apply(words: Seq[String]): Seq[Int]

The application of this Transformer to a single input item.
The application of this Transformer to a single input item. This method MUST be overridden by ML developers.
returns
The output value

Definition Classes
WordFrequencyTransformer → Transformer
def apply(in: RDD[Seq[String]]): RDD[Seq[Int]]

The application of this Transformer to an RDD of input items.
The application of this Transformer to an RDD of input items. This method may optionally be overridden by ML developers.
in
The bulk RDD input to pass into this transformer
returns
The bulk RDD output for the given input

Definition Classes
WordFrequencyTransformer → Transformer
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def execute(deps: Seq[Expression]): Expression

Definition Classes
TransformerOperator → Operator
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def label: String

Definition Classes
Operator
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toPipeline: Pipeline[Seq[String], Seq[Int]]

A method that converts this object into a Pipeline.
A method that converts this object into a Pipeline. Must be implemented by anything that extends Chainable.

Definition Classes
Transformer → Chainable
def toString(): String

Definition Classes
AnyRef → Any
val unigramCounts: Map[Int, Int]

the counts of unigrams in the training corpus
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

class WordFrequencyTransformer extends Transformer[Seq[String], Seq[Int]]

Instance Constructors

new WordFrequencyTransformer(wordIndexBroadcast: Broadcast[Map[String, Int]], unigramCounts: Map[Int, Int])

Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

final val OOV_INDEX: Int(-1)

final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: PipelineDataset[Seq[String]], labels: PipelineDataset[L]): Pipeline[Seq[String], C]

final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: RDD[Seq[String]], labels: PipelineDataset[L]): Pipeline[Seq[String], C]

final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: PipelineDataset[Seq[String]], labels: RDD[L]): Pipeline[Seq[String], C]

final def andThen[C, L](est: LabelEstimator[Seq[Int], C, L], data: RDD[Seq[String]], labels: RDD[L]): Pipeline[Seq[String], C]

final def andThen[C](est: Estimator[Seq[Int], C], data: PipelineDataset[Seq[String]]): Pipeline[Seq[String], C]

final def andThen[C](est: Estimator[Seq[Int], C], data: RDD[Seq[String]]): Pipeline[Seq[String], C]

final def andThen[C](next: Chainable[Seq[Int], C]): Pipeline[Seq[String], C]

def apply(words: Seq[String]): Seq[Int]

def apply(in: RDD[Seq[String]]): RDD[Seq[Int]]

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def execute(deps: Seq[Expression]): Expression

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

def label: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toPipeline: Pipeline[Seq[String], Seq[Int]]

def toString(): String

val unigramCounts: Map[Int, Int]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Transformer[Seq[String], Seq[Int]]

Inherited from Chainable[Seq[String], Seq[Int]]

Inherited from TransformerOperator

Inherited from Serializable

Inherited from Serializable

Inherited from Operator

Inherited from AnyRef

Inherited from Any

Ungrouped