[SPARK-14585][ML][WIP] Provide accessor methods for Pipeline stages

[pravingadakh]

What changes were proposed in this pull request?

This PR adds accessor methods for pipeline stages in Pipeline and PipelineModel classes.

---

@jkbradley Hi, please validate if this is something that you had in mind. If so, I can start on test cases for these.

[jkbradley]

test this please

[SparkQA]

Test build #56401 has finished for PR 12420 at commit a4592c6.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

@pravingadakh Thanks! This is what I had in mind. In your tests, could you please add Java unit tests too to make sure these are Java-friendly?

Also, please check the type and throw exceptions when the type does not match. I'd recommend throwing IllegalArgumentException and specifying the expected and actual types.

[jkbradley]

Here and elsewhere, I would just write array(i) instead of array.apply(i)

Also, can you please document the @tparam?

[pravingadakh]

@jkbradley I'll update the PR accordingly.

[jkbradley]

By the way, we'll need to get this merged quickly for it to make 2.0. (Thanks!)

[pravingadakh]

@jkbradley I'll update it soon.

[pravingadakh]

@jkbradley I have stumbled across an issue, would really appreciate your input. Have a look at this method:

def getStage[T <: PipelineStage : ClassTag](i: Int): T = {
    getStages(i) match {
      case stage: T => stage
      case stage => throw new IllegalArgumentException(
        s"Expected stage of type ${stage.getClass.getName}"
      )
    }
  }

How do I avoid usage of ClassTag here (since those are not so java friendly)? Also I'm a bit skeptical about the error message.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Can't be since 2.0.0 at this point. Also use a @return tag in the docs.

[srowen]

Nit: This is probably more natural as a one-liner

[jkbradley]

@pravingadakh Will you be able to continue with this? Thanks!

[jkbradley]

I missed the ClassTag question above. Let me take a look

[jkbradley]

Well, after spending a while looking around, I haven't found a good way to write this and make it Java friendly (i.e., not use ClassTag, Type, or TypeTag). Does anyone else have ideas? I'll try to return to this before long...

[HyukjinKwon]

Hi @@pravingadakh, is this still active?