device agnostic models testing #27146

ji-huazhong · 2023-10-30T12:47:48Z

What does this PR do?

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

amyeroberts · 2023-10-30T13:36:05Z

Thanks for opening this PR @statelesshz ! Let us know when you'd like a review

ji-huazhong · 2023-10-31T07:44:30Z

Due to network issue， it's not easy to download the model weights on huggingface hub from my area, so I take test_modeling_bert.py as an example to give the execution reaults on NPU.

(hf) [root@localhost /home/hf/transformers]# RUN_SLOW=1 TRANSFORMERS_TEST_BACKEND="torch_npu" TRANSFORMERS_TEST_DEVICE="npu:0" TRANSFORMEs/bert/test_modeling_bert.py
========================================================================================= test session starts ===========================
platform linux -- Python 3.8.17, pytest-7.4.2, pluggy-1.3.0 -- /root/anaconda3/envs/hf/bin/python
cachedir: .pytest_cache
rootdir: /home/hf/transformers
configfile: setup.cfg
plugins: dash-2.13.0, hydra-core-1.3.2, odl-0.7.0, anyio-4.0.0
collected 135 items                                                                                                                      

tests/models/bert/test_modeling_bert.py::BertModelTest::test_assisted_decoding_matches_greedy_search PASSED                              
tests/models/bert/test_modeling_bert.py::BertModelTest::test_assisted_decoding_sample PASSED                                             
tests/models/bert/test_modeling_bert.py::BertModelTest::test_attention_outputs PASSED                                                    
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_sample_generate PASSED                                                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_sample_generate_dict_output PASSED                                     
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_search_generate PASSED                                                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_search_generate_dict_output PASSED                                     
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_search_generate_dict_outputs_use_cache PASSED                          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_can_use_safetensors PASSED                                                  
tests/models/bert/test_modeling_bert.py::BertModelTest::test_config PASSED                                                               
tests/models/bert/test_modeling_bert.py::BertModelTest::test_constrained_beam_search_generate SKIPPED (unconditional skip)               
tests/models/bert/test_modeling_bert.py::BertModelTest::test_constrained_beam_search_generate_dict_output SKIPPED (unconditional skip)   
tests/models/bert/test_modeling_bert.py::BertModelTest::test_contrastive_generate PASSED                                                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_contrastive_generate_dict_outputs_use_cache PASSED                          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_contrastive_generate_low_memory PASSED                                      
tests/models/bert/test_modeling_bert.py::BertModelTest::test_correct_missing_keys PASSED                                                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_cpu_offload SKIPPED (test requires CUDA)                                    
tests/models/bert/test_modeling_bert.py::BertModelTest::test_decoder_model_past_with_large_inputs PASSED                                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_decoder_model_past_with_large_inputs_relative_pos_emb PASSED                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_determinism PASSED                                                          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_disk_offload SKIPPED (test requires CUDA)                                   
tests/models/bert/test_modeling_bert.py::BertModelTest::test_equivalence_flax_to_pt SKIPPED (test is PT+FLAX test)                       
tests/models/bert/test_modeling_bert.py::BertModelTest::test_equivalence_pt_to_flax SKIPPED (test is PT+FLAX test)                       
tests/models/bert/test_modeling_bert.py::BertModelTest::test_feed_forward_chunking PASSED                                                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_flash_attn_2_conversion SKIPPED (test requires Flash Attention)             
tests/models/bert/test_modeling_bert.py::BertModelTest::test_flash_attn_2_fp32_ln SKIPPED (test requires Flash Attention)                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_flash_attn_2_generate_left_padding SKIPPED (test requires Flash Attention)  
tests/models/bert/test_modeling_bert.py::BertModelTest::test_flash_attn_2_generate_padding_right SKIPPED (test requires Flash Attention) 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_flash_attn_2_generate_use_cache SKIPPED (test requires Flash Attention)     
tests/models/bert/test_modeling_bert.py::BertModelTest::test_flash_attn_2_inference SKIPPED (test requires Flash Attention)              
tests/models/bert/test_modeling_bert.py::BertModelTest::test_flash_attn_2_inference_padding_right SKIPPED (test requires Flash Attention)
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_causal_lm PASSED                                                        
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_causal_lm_decoder PASSED                                                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_masked_lm PASSED                                                        
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_multiple_choice PASSED                                                  
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_next_sequence_prediction PASSED                                         
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_pretraining PASSED                                                      
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_question_answering PASSED                                               
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_sequence_classification PASSED                                          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_token_classification PASSED                                             
tests/models/bert/test_modeling_bert.py::BertModelTest::test_for_warning_if_padding_and_no_attention_mask PASSED                         
tests/models/bert/test_modeling_bert.py::BertModelTest::test_forward_signature PASSED                                                    
tests/models/bert/test_modeling_bert.py::BertModelTest::test_from_pretrained_no_checkpoint PASSED                                        
tests/models/bert/test_modeling_bert.py::BertModelTest::test_generate_from_inputs_embeds_decoder_only PASSED                             
tests/models/bert/test_modeling_bert.py::BertModelTest::test_generate_with_head_masking PASSED                                           
tests/models/bert/test_modeling_bert.py::BertModelTest::test_generate_without_input_ids PASSED                                           
tests/models/bert/test_modeling_bert.py::BertModelTest::test_gradient_checkpointing_backward_compatibility PASSED                        
tests/models/bert/test_modeling_bert.py::BertModelTest::test_gradient_checkpointing_enable_disable PASSED                                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_greedy_generate PASSED                                                      
tests/models/bert/test_modeling_bert.py::BertModelTest::test_greedy_generate_dict_outputs PASSED                                         
tests/models/bert/test_modeling_bert.py::BertModelTest::test_greedy_generate_dict_outputs_use_cache PASSED                               
tests/models/bert/test_modeling_bert.py::BertModelTest::test_group_beam_search_generate PASSED                                           
tests/models/bert/test_modeling_bert.py::BertModelTest::test_group_beam_search_generate_dict_output PASSED                               
tests/models/bert/test_modeling_bert.py::BertModelTest::test_head_pruning PASSED                                                         
tests/models/bert/test_modeling_bert.py::BertModelTest::test_head_pruning_integration PASSED                                             
tests/models/bert/test_modeling_bert.py::BertModelTest::test_head_pruning_save_load_from_config_init PASSED                              
tests/models/bert/test_modeling_bert.py::BertModelTest::test_head_pruning_save_load_from_pretrained PASSED                               
tests/models/bert/test_modeling_bert.py::BertModelTest::test_headmasking PASSED                                                          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_hidden_states_output PASSED                                                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_initialization PASSED                                                       
tests/models/bert/test_modeling_bert.py::BertModelTest::test_inputs_embeds PASSED                                                        
tests/models/bert/test_modeling_bert.py::BertModelTest::test_keep_in_fp32_modules PASSED                                                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_left_padding_compatibility PASSED                                           
tests/models/bert/test_modeling_bert.py::BertModelTest::test_load_save_without_tied_weights PASSED                                       
tests/models/bert/test_modeling_bert.py::BertModelTest::test_load_with_mismatched_shapes PASSED                                          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model PASSED                                                                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_as_decoder PASSED                                                     
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_as_decoder_with_default_input_mask PASSED                             
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_common_attributes PASSED                                              
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_from_pretrained PASSED                                                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_is_small PASSED                                                       
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_main_input_name PASSED                                                
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_outputs_equivalence PASSED                                            
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_parallel_beam_search PASSED                                           
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_parallel_equal_results SKIPPED (test requires multiple GPUs)          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_parallelism SKIPPED (test requires multiple GPUs)                     
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_parallelization SKIPPED (test requires multiple GPUs)                 
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_various_embeddings PASSED                                             
tests/models/bert/test_modeling_bert.py::BertModelTest::test_model_weights_reload_no_missing_tied_weights PASSED                         
tests/models/bert/test_modeling_bert.py::BertModelTest::test_multi_gpu_data_parallel_forward SKIPPED (test requires multiple GPUs)       
tests/models/bert/test_modeling_bert.py::BertModelTest::test_past_key_values_format PASSED                                               
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_audio_classification SKIPPED (BertModelTest::test_pipeline_audio_cl
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_automatic_speech_recognition SKIPPED (BertModelTest::test_pipeline_
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_conversational SKIPPED (BertModelTest::test_pipeline_conversational
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_depth_estimation SKIPPED (BertModelTest::test_pipeline_depth_estima
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_document_question_answering SKIPPED (test requires PyTesseract)    
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_feature_extraction PASSED                                          

                                                                               
PASSED                                                                                                          [ 65%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_image_classification SKIPPED (BertModelTest::test_pipeline_image_cl
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_image_segmentation SKIPPED (BertModelTest::test_pipeline_image_segm
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_image_to_text SKIPPED (BertModelTest::test_pipeline_image_to_text i
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_mask_generation SKIPPED (`run_pipeline_test` is currently not imple
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_object_detection SKIPPED (BertModelTest::test_pipeline_object_detec
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_question_answering PASSED                                          
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_summarization SKIPPED (BertModelTest::test_pipeline_summarization i
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_table_question_answering SKIPPED (BertModelTest::test_pipeline_tabl
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_text2text_generation SKIPPED (BertModelTest::test_pipeline_text2tex
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_text_classification PASSED                                         
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_text_generation 


PASSED                                     [ 73%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_text_to_audio SKIPPED (BertModelTest::test_pipeline_te...) [ 74%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_token_classification PASSED                                [ 74%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_translation SKIPPED (BertModelTest::test_pipeline_tran...) [ 75%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_video_classification SKIPPED (test requires decord)        [ 76%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_visual_question_answering SKIPPED (BertModelTest::test...) [ 77%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot PASSED                                           [ 77%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot_audio_classification SKIPPED (BertModelTest:...) [ 78%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot_image_classification SKIPPED (BertModelTest:...) [ 79%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot_object_detection SKIPPED (BertModelTest::tes...) [ 80%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_problem_types PASSED                                                [ 80%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_pt_tf_model_equivalence SKIPPED (test is PT+TF test)                [ 81%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_resize_embeddings_untied PASSED                                     [ 82%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_resize_position_vector_embeddings PASSED                            [ 82%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_resize_tokens_embeddings PASSED                                     [ 83%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_retain_grad_hidden_states_attentions PASSED                         [ 84%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_sample_generate PASSED                                              [ 85%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_sample_generate_dict_output PASSED                                  [ 85%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_save_load PASSED                                                    [ 86%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_save_load_fast_init_from_base PASSED                                [ 87%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_save_load_fast_init_to_base PASSED                                  [ 88%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_save_load_keys_to_ignore_on_save PASSED                             [ 88%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_tie_model_weights PASSED                                            [ 89%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_tied_weights_keys PASSED                                            [ 90%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_torch_fx PASSED                                                     [ 91%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_torch_fx_output_loss PASSED                                         [ 91%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_device_change PASSED                                    [ 92%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_output_attentions PASSED                                [ 93%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_output_hidden_state PASSED                              [ 94%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_simple PASSED                                           [ 94%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_training PASSED                                                     [ 95%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_training_gradient_checkpointing PASSED                              [ 96%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_training_gradient_checkpointing_use_reentrant PASSED                [ 97%]
tests/models/bert/test_modeling_bert.py::BertModelTest::test_training_gradient_checkpointing_use_reentrant_false PASSED          [ 97%]
tests/models/bert/test_modeling_bert.py::BertModelIntegrationTest::test_inference_no_head_relative_embedding_key PASSED          [ 99%]
tests/models/bert/test_modeling_bert.py::BertModelIntegrationTest::test_inference_no_head_relative_embedding_key_query PASSED    [100%]

=========================================================== warnings summary ===========================================================
../../../root/anaconda3/envs/hf/lib/python3.8/site-packages/_pytest/config/__init__.py:1373
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/test_modeling_common.py:2790
  /home/hf/transformers/tests/test_modeling_common.py:2790: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2817
  /home/hf/transformers/tests/test_modeling_common.py:2817: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2863
  /home/hf/transformers/tests/test_modeling_common.py:2863: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2905
  /home/hf/transformers/tests/test_modeling_common.py:2905: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2942
  /home/hf/transformers/tests/test_modeling_common.py:2942: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2979
  /home/hf/transformers/tests/test_modeling_common.py:2979: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:3009
  /home/hf/transformers/tests/test_modeling_common.py:3009: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

src/transformers/deepspeed.py:23
  /home/hf/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_assisted_decoding_matches_greedy_search
  /home/hf/transformers/tests/test_modeling_common.py:3069: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/_internal/cpython-3.8.17/lib/python3.8/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h:74.)
    attn_mask[:, 0] = 1

tests/models/bert/test_modeling_bert.py::BertModelTest::test_attention_outputs
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch_npu/utils/storage.py:36: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    if self.device.type != 'cpu':

tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_sample_generate
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_sample_generate_dict_output
  /home/hf/transformers/src/transformers/generation/utils.py:3341: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_search_generate
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_search_generate_dict_output
tests/models/bert/test_modeling_bert.py::BertModelTest::test_beam_search_generate_dict_outputs_use_cache
  /home/hf/transformers/src/transformers/generation/utils.py:3005: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_greedy_generate
tests/models/bert/test_modeling_bert.py::BertModelTest::test_greedy_generate_dict_outputs
tests/models/bert/test_modeling_bert.py::BertModelTest::test_greedy_generate_dict_outputs_use_cache
  /home/hf/transformers/src/transformers/generation/utils.py:2450: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList([MaxLengthCriteria(max_length=max_length)])` instead.
    warnings.warn(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_group_beam_search_generate
tests/models/bert/test_modeling_bert.py::BertModelTest::test_group_beam_search_generate_dict_output
  /home/hf/transformers/src/transformers/generation/utils.py:3663: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/models/bert/test_modeling_bert.py: 54 warnings
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '90.253.31.68'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
    warnings.warn(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_text_generation
  /home/hf/transformers/src/transformers/generation/utils.py:1273: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.
    warnings.warn(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py:642: UserWarning: Length of IterableDataset <transformers.pipelines.pt_utils.PipelineChunkIterator object at 0xffff395d15e0> was reported to be 1 (when accessing len(dataloader)), but 2 samples have been fetched. 
    warnings.warn(warn_msg)

tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py:642: UserWarning: Length of IterableDataset <transformers.pipelines.pt_utils.PipelineChunkIterator object at 0xffff395d1790> was reported to be 2 (when accessing len(dataloader)), but 3 samples have been fetched. 
    warnings.warn(warn_msg)

tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py:642: UserWarning: Length of IterableDataset <transformers.pipelines.pt_utils.PipelineChunkIterator object at 0xffff395d1790> was reported to be 2 (when accessing len(dataloader)), but 4 samples have been fetched. 
    warnings.warn(warn_msg)

tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py:642: UserWarning: Length of IterableDataset <transformers.pipelines.pt_utils.PipelineChunkIterator object at 0xffff395be6a0> was reported to be 1 (when accessing len(dataloader)), but 2 samples have been fetched. 
    warnings.warn(warn_msg)

tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py:642: UserWarning: Length of IterableDataset <transformers.pipelines.pt_utils.PipelineChunkIterator object at 0xffff395d10a0> was reported to be 2 (when accessing len(dataloader)), but 3 samples have been fetched. 
    warnings.warn(warn_msg)

tests/models/bert/test_modeling_bert.py::BertModelTest::test_pipeline_zero_shot
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py:642: UserWarning: Length of IterableDataset <transformers.pipelines.pt_utils.PipelineChunkIterator object at 0xffff395d10a0> was reported to be 2 (when accessing len(dataloader)), but 4 samples have been fetched. 
    warnings.warn(warn_msg)

tests/models/bert/test_modeling_bert.py::BertModelTest::test_sample_generate
tests/models/bert/test_modeling_bert.py::BertModelTest::test_sample_generate_dict_output
  /home/hf/transformers/src/transformers/generation/utils.py:2728: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torch_fx
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/overrides.py:110: UserWarning: 'has_cuda' is deprecated, please use 'torch.backends.cuda.is_built()'
    torch.has_cuda,

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torch_fx
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/overrides.py:111: UserWarning: 'has_cudnn' is deprecated, please use 'torch.backends.cudnn.is_available()'
    torch.has_cudnn,

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torch_fx
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/overrides.py:117: UserWarning: 'has_mps' is deprecated, please use 'torch.backends.mps.is_built()'
    torch.has_mps,

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torch_fx
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/overrides.py:118: UserWarning: 'has_mkldnn' is deprecated, please use 'torch.backends.mkldnn.is_available()'
    torch.has_mkldnn,

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_device_change
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/jit/_trace.py:1093: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
  Tensor-likes are not close!
  
  Mismatched elements: 2911 / 2912 (100.0%)
  Greatest absolute difference: 2.786480948328972 at index (10, 5, 26) (up to 1e-05 allowed)
  Greatest relative difference: 945.6015975578661 at index (6, 3, 1) (up to 1e-05 allowed)
    _check_trace(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_device_change
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/jit/_trace.py:1093: TracerWarning: Output nr 2. of the traced function does not match the corresponding output of the Python function. Detailed error:
  Tensor-likes are not close!
  
  Mismatched elements: 416 / 416 (100.0%)
  Greatest absolute difference: 0.20477154292166233 at index (4, 21) (up to 1e-05 allowed)
  Greatest relative difference: 46.22814215551245 at index (11, 11) (up to 1e-05 allowed)
    _check_trace(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_device_change
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/jit/_trace.py:1093: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
  Tensor-likes are not close!
  
  Mismatched elements: 8917 / 9009 (99.0%)
  Greatest absolute difference: 0.27886080741882324 at index (3, 4, 13) (up to 1e-05 allowed)
  Greatest relative difference: 5961.7773065868605 at index (5, 1, 61) (up to 1e-05 allowed)
    _check_trace(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_torchscript_device_change
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/jit/_trace.py:1093: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
  Tensor-likes are not close!
  
  Mismatched elements: 8916 / 9009 (99.0%)
  Greatest absolute difference: 0.33901818469166756 at index (5, 4, 33) (up to 1e-05 allowed)
  Greatest relative difference: 8042.04489700118 at index (11, 5, 78) (up to 1e-05 allowed)
    _check_trace(

tests/models/bert/test_modeling_bert.py::BertModelTest::test_training_gradient_checkpointing
  /root/anaconda3/envs/hf/lib/python3.8/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================= 97 passed, 38 skipped, 93 warnings in 1067.07s (0:17:47) ======================================

ji-huazhong · 2023-10-31T07:45:57Z

Marking this ready for review :-) @ydshieh and @amyeroberts

amyeroberts

Another great PR - thank you!

Happy to merge once @ydshieh gives his 👍

amyeroberts · 2023-10-31T10:52:05Z

tests/models/bark/test_modeling_bark.py

        model = self.all_generative_model_classes[0](config).eval().to(torch_device)
-        if torch_device == "cuda":
-            model.half()
+        model.half()


So much cleaner 🤩

tests/models/gptj/test_modeling_gptj.py

ydshieh · 2023-10-31T16:24:57Z

tests/models/jukebox/test_modeling_jukebox.py

    @slow
-    @require_torch_gpu
+    @require_torch_accelerator
+    @require_torch_fp16


why fp16 here?

Oops, the fp16 decorator was misused here. Thanks for pointing it out

ydshieh · 2023-10-31T16:35:58Z

Nice to know bert works well on NPU!

ydshieh · 2023-10-31T17:09:37Z

Thank you @statelesshz the king of NPU!

HuggingFaceDocBuilderDev · 2023-10-31T17:12:24Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

* device agnostic models testing * add decorator `require_torch_fp16` * make style * apply review suggestion * Oops, the fp16 decorator was misused

ji-huazhong marked this pull request as draft October 30, 2023 13:35

ji-huazhong changed the title ~~device agnostic models testing~~ [WIP]device agnostic models testing Oct 30, 2023

ji-huazhong force-pushed the device-agnostic-models-testing branch from fa43b77 to 28aef47 Compare October 31, 2023 02:19

ji-huazhong added 2 commits October 31, 2023 10:39

device agnostic models testing

0c4e59b

add decorator require_torch_fp16

443c50c

ji-huazhong force-pushed the device-agnostic-models-testing branch from 28aef47 to 9d37036 Compare October 31, 2023 02:40

make style

e94cc76

ji-huazhong force-pushed the device-agnostic-models-testing branch from 9d37036 to e94cc76 Compare October 31, 2023 02:54

ji-huazhong changed the title ~~[WIP]device agnostic models testing~~ device agnostic models testing Oct 31, 2023

ji-huazhong mentioned this pull request Oct 31, 2023

Make 🤗Transformers tests device agnostic #25654

Open

ji-huazhong marked this pull request as ready for review October 31, 2023 07:40

amyeroberts approved these changes Oct 31, 2023

View reviewed changes

ydshieh approved these changes Oct 31, 2023

View reviewed changes

ji-huazhong added 2 commits November 1, 2023 00:42

apply review suggestion

14c32ad

Oops, the fp16 decorator was misused

f0762cc

ydshieh merged commit 50378cb into huggingface:main Oct 31, 2023

ji-huazhong deleted the device-agnostic-models-testing branch November 1, 2023 06:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

device agnostic models testing #27146

device agnostic models testing #27146

Uh oh!

ji-huazhong commented Oct 30, 2023 •

edited

Loading

Uh oh!

amyeroberts commented Oct 30, 2023

Uh oh!

ji-huazhong commented Oct 31, 2023

Uh oh!

ji-huazhong commented Oct 31, 2023

Uh oh!

amyeroberts left a comment

Uh oh!

amyeroberts Oct 31, 2023

Uh oh!

Uh oh!

ydshieh Oct 31, 2023

Uh oh!

ji-huazhong Oct 31, 2023

Uh oh!

ydshieh commented Oct 31, 2023

Uh oh!

ydshieh commented Oct 31, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Oct 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

device agnostic models testing #27146

device agnostic models testing #27146

Uh oh!

Conversation

ji-huazhong commented Oct 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

amyeroberts commented Oct 30, 2023

Uh oh!

ji-huazhong commented Oct 31, 2023

Uh oh!

ji-huazhong commented Oct 31, 2023

Uh oh!

amyeroberts left a comment

Choose a reason for hiding this comment

Uh oh!

amyeroberts Oct 31, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ydshieh Oct 31, 2023

Choose a reason for hiding this comment

Uh oh!

ji-huazhong Oct 31, 2023

Choose a reason for hiding this comment

Uh oh!

ydshieh commented Oct 31, 2023

Uh oh!

ydshieh commented Oct 31, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Oct 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ji-huazhong commented Oct 30, 2023 •

edited

Loading