Skip to content

Fix smoother imputing polynomial order #490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Nov 11, 2020
Merged

Fix smoother imputing polynomial order #490

merged 4 commits into from
Nov 11, 2020

Conversation

@dshemetov
Copy link
Contributor

@dshemetov dshemetov commented Nov 10, 2020 •
edited
Loading

Description

Update the smoother's imputation function to perform better. Depends on #478 .

Changelog

  • Changes the smoother imputation function to have its own polynomial order than the smoother's
  • The default value for the imputer is also set to 2.

Fixes

  • Arguably the imputer should interpolate values instead of smoothing them. Hence we avoid situation like this:
>>> signal = np.array([i if i % 3 else np.nan for i in range(1, 40)])
>>> Smoother().impute(signal, impute_order=0)
array([ 1.        ,  2.        ,  1.50520814,  4.        ,  5.        ,
        2.77986492,  7.        ,  8.        ,  4.19921816, 10.        ,
       11.        ,  5.82790041, 13.        , 14.        ,  7.70752267,
       16.        , 17.        ,  9.84727107, 19.        , 20.        ,
       12.22315472, 22.        , 23.        , 14.78905031, 25.        ,
       26.        , 17.4936213 , 28.        , 29.        , 20.29881603,
       31.        , 32.        , 23.17118576, 34.        , 35.        ,
       26.08145499, 37.        , 38.        , 29.01928305])
>>> Smoother().impute(signal, impute_order=2)
array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
       14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
       27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.])

Without this change, using poly_fit_degree=0 would use this setting in the imputer and introduce the poor performance above.

@dshemetov dshemetov requested a review from chinandrew November 10, 2020 23:10
chinandrew
chinandrew reviewed Nov 10, 2020
View reviewed changes
_delphi_utils_python/delphi_utils/smooth.py Outdated
)
# Otherwise, use savgol fitting on the largest window prior
# Otherwise, use savgol fitting on the largest window prior,
# reduce the polynomial degree if needed
Copy link
Contributor

@chinandrew chinandrew Nov 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a note why the reduction of the polynomial degree is needed.

Copy link
Contributor Author

@dshemetov dshemetov Nov 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

chinandrew
chinandrew reviewed Nov 10, 2020
View reviewed changes
Copy link
Contributor

@chinandrew chinandrew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one comment otherwise lgtm

@dshemetov
Updated the smoother to remove a non-invertible design matrix bug
a5f81ba
@dshemetov
Update smoother to gracefully handle nans:
4ad4c39 
* entire array of nans is handled
* left-padded nans are now ignored
* a few other edge cases
* add tests to match
@dshemetov
Fixes the bug where the smoother drops the pandas series index:
ad01388 
* restore the index after smoothing
* test to match
@dshemetov dshemetov force-pushed the fix_smoother_imputing_polyorder branch from a5c18a1 to 45ab905 Compare November 11, 2020 00:20
@dshemetov
Update smoother imputer:
fa0b6e2 
* separate out the smoother's polynomial fit degree from the imputer's
* default the imputer's fit degree to 2
* add tests
@dshemetov dshemetov force-pushed the fix_smoother_imputing_polyorder branch from 45ab905 to fa0b6e2 Compare November 11, 2020 00:23
chinandrew
chinandrew approved these changes Nov 11, 2020
View reviewed changes
Copy link
Contributor

@chinandrew chinandrew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit: :shipit: :shipit:

@chinandrew
Copy link
Contributor

chinandrew commented Nov 11, 2020

This depends on #476 -8 right? maybe should add a section for that on the pr template....

@dshemetov
Copy link
Contributor Author

dshemetov commented Nov 11, 2020

Oops, meant to do that

@krivard krivard merged commit 95ece40 into main Nov 11, 2020
@krivard krivard deleted the fix_smoother_imputing_polyorder branch November 11, 2020 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chinandrew chinandrew chinandrew approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dshemetov @chinandrew @krivard