[SPARK-19939] [ML] Add support for association rules in ML #17280

hhbyyh · 2017-03-13T18:26:55Z

What changes were proposed in this pull request?

jira: https://issues.apache.org/jira/browse/SPARK-19939
Adding another essential characteristic for the Association Rule in Spark ml.fpm.

Support is an indication of how frequently the itemset of an association rule appears in the database and suggests if the rules are generally applicable to the dateset. refer to wiki for more details.

Before adding support:

rules	confidence
beer -> soda	0.5
pecan -> milk	0.6

After adding support:

rules	confidence	support
beer -> soda	0.5	0.3
pecan -> milk	0.6	0.01

Thus to allow a better understanding for the generated association rules. This is a new feature and was not included in the original function parity PR.

How was this patch tested?

existing and new unit test

SparkQA · 2017-03-13T19:19:58Z

Test build #74458 has finished for PR 17280 at commit 9580aa9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hhbyyh · 2017-03-15T17:15:54Z

I'll focus on #17130 first and resolve conflict here.

SparkQA · 2017-03-15T18:16:51Z

Test build #74610 has finished for PR 17280 at commit 1225496.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-11T21:01:06Z

Test build #75715 has finished for PR 17280 at commit 3f780dc.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

hhbyyh · 2017-04-17T18:20:02Z

I'll update this after FPGrowth examples and doc merged #17130, since there'll be some conflicts.

update: Done merging the conflict

-better code

SparkQA · 2017-06-09T00:39:56Z

Test build #77829 has finished for PR 17280 at commit 180b43d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-13T21:22:40Z

Test build #78005 has started for PR 17280 at commit eb25f28.

shaneknapp · 2017-06-13T23:19:05Z

test this please

SparkQA · 2017-06-14T00:29:25Z

Test build #78010 has finished for PR 17280 at commit eb25f28.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2017-07-01T04:28:59Z

python/pyspark/ml/fpm.py

+    |   [r]|   3|
+    |   [y]|   3|
+    |[y, x]|   3|
+    +------+----+


this seems to change the result quite a bit, is this expected?

or rather, why is https://github.com/apache/spark/pull/17280/files#diff-b6dbf16870bd2cca9b4140df8aebd681L189 changed?

set a higher minSupport to avoid the 0.3333333... in the support column.

MLnick · 2018-01-16T07:51:45Z

mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala

    override def load(path: String): FPGrowthModel = {
      val metadata = DefaultParamsReader.loadMetadata(path, sc, className)
+      implicit val format = DefaultFormats
+      val numTrainingRecords = (metadata.metadata \ "numTrainingRecords").extract[Long]


Does this break backward compatibility for loading?

Yes it does now. If saving numTrainingRecords to FPGrowthModel looks good, I can update the load logic.

Since we're adding numTrainingRecords to FPGrowthModel and there isn't a proper default number, I suggest we break the backward model loading compatibility. Otherwise we need to fill numTrainingRecords with an incorrect value and will likely create a maintenance trap.

hhbyyh · 2018-01-17T05:59:02Z

Thanks for taking a look @MLnick

hhbyyh · 2018-03-21T03:55:55Z

Please advice if this is a good feature to add. If not I'll close it. Thanks.

SparkQA · 2018-07-01T15:08:53Z

Test build #92514 has finished for PR 17280 at commit ce6959a.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-30T23:24:26Z

Test build #93814 has finished for PR 17280 at commit 40cf449.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2018-08-01T01:00:47Z

Test build #93842 has finished for PR 17280 at commit 34f883f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hhbyyh · 2018-08-01T03:03:51Z

Updated to support backward model loading compatibility.
@MLnick @jkbradley

SparkQA · 2018-08-27T22:04:29Z

Test build #95316 has finished for PR 17280 at commit 9e2854a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-28T02:51:51Z

Test build #95318 has finished for PR 17280 at commit 733c7ff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2020-02-01T00:05:24Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

gungor2 · 2021-04-20T15:08:38Z

Is there any plan to revive this PR? It would be great if we can the support of rules exposed.

YY-OnCall added 4 commits March 10, 2017 17:18

half

f86c512

Merge remote-tracking branch 'upstream/master' into arSupport

bb8ed11

Merge remote-tracking branch 'upstream/master' into arSupport

a0ad662

add support to association rules

9580aa9

hhbyyh mentioned this pull request Mar 14, 2017

[SPARK-15938]Adding "support" property to MLlib Association Rule #13656

Closed

YY-OnCall added 2 commits March 14, 2017 09:15

Merge remote-tracking branch 'upstream/master' into arSupport

fabb194

update comment

1225496

resolve conflict

3f780dc

garis added a commit to garis/DataMiningBackBlazeCustomSpark that referenced this pull request May 2, 2017

-spark with apache/spark#17280

9b99d18

-better code

YY-OnCall added 2 commits June 8, 2017 15:52

merge conflict

670ab27

merge conflict

180b43d

YY-OnCall added 2 commits June 13, 2017 11:33

Merge remote-tracking branch 'upstream/master' into arSupport

f5959ae

fix python and R test

eb25f28

felixcheung reviewed Jul 1, 2017

View reviewed changes

Merge remote-tracking branch 'upstream/master' into arSupport

ba48226

MLnick reviewed Jan 16, 2018

View reviewed changes

Merge remote-tracking branch 'upstream/master' into arSupport

2c008b7

resolve conflict

ce6959a

mima fix

40cf449

resolve conflict

34f883f

merge conflict

9e2854a

comma

733c7ff

hhbyyh mentioned this pull request Aug 28, 2018

[SPARK-10697][ML] Add lift to Association rules #22236

Closed

dongjoon-hyun added the ML label Jun 14, 2019

github-actions bot added the Stale label Feb 1, 2020

github-actions bot closed this Feb 2, 2020

[SPARK-19939] [ML] Add support for association rules in ML #17280

[SPARK-19939] [ML] Add support for association rules in ML #17280

Uh oh!

Conversation

hhbyyh commented Mar 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

hhbyyh commented Mar 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Mar 15, 2017

Uh oh!

SparkQA commented Apr 11, 2017

Uh oh!

hhbyyh commented Apr 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jun 9, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

shaneknapp commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 14, 2017

Uh oh!

felixcheung Jul 1, 2017

Choose a reason for hiding this comment

Uh oh!

felixcheung Jul 1, 2017

Choose a reason for hiding this comment

Uh oh!

hhbyyh Jul 5, 2017

Choose a reason for hiding this comment

Uh oh!

MLnick Jan 16, 2018

Choose a reason for hiding this comment

Uh oh!

hhbyyh Jan 17, 2018

Choose a reason for hiding this comment

Uh oh!

hhbyyh Jan 31, 2018

Choose a reason for hiding this comment

Uh oh!

hhbyyh commented Jan 17, 2018

Uh oh!

hhbyyh commented Mar 21, 2018

Uh oh!

SparkQA commented Jul 1, 2018

Uh oh!

SparkQA commented Jul 30, 2018

Uh oh!

SparkQA commented Aug 1, 2018

Uh oh!

hhbyyh commented Aug 1, 2018

Uh oh!

SparkQA commented Aug 27, 2018

Uh oh!

SparkQA commented Aug 28, 2018

Uh oh!

github-actions bot commented Feb 1, 2020

Uh oh!

gungor2 commented Apr 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

hhbyyh commented Mar 13, 2017 •

edited

Loading

hhbyyh commented Mar 15, 2017 •

edited

Loading

hhbyyh commented Apr 17, 2017 •

edited

Loading