Inference regions (#338)

ESegundoRolon · Enrique · crmne · web-flow · commit 791f28b60807 · 2025-09-06T20:03:40.000+02:00
## What this does Support region-aware inference profiles for AWS Bedrock ### Problem The current implementation hardcodes us. prefix for all Bedrock inference profile models, causing failures when using cross-region inference in non-US regions like EU, AP, or CA. - Error encountered: The provided model identifier is invalid. (RubyLLM::BadRequestError) This occurs because: 1. EU regions expect for example `eu.anthropic.claude-sonnet-4-20250514-v1:0` 2. But the library sends `us.anthropic.claude-sonnet-4-20250514-v1:0` 3. AWS Bedrock rejects the incorrect region prefix ### Solution Updated model_id_with_region method to dynamically extract the region prefix from the configured bedrock_region: - eu-west-3 → eu. prefix - us-east-1 → us. prefix - ap-south-1 → ap. prefix - ca-central-1 → ca. prefix ### Changes Made Core Fix File: lib/ruby_llm/providers/bedrock/models.rb - Modified model_id_with_region to use dynamic region prefix - Added inference_profile_region_prefix helper method - Uses first two characters of bedrock_region as prefix - Maintains backwards compatibility with us default Tests Added File: spec/ruby_llm/providers/bedrock/models_spec.rb - Comprehensive tests for different AWS regions (US, EU, AP, CA) - Tests for edge cases (empty/nil regions) - Tests for both inference profile and on-demand models - Ensures backwards compatibility Testing - ✅ US regions continue to work (us. prefix) - ✅ EU regions now work (eu. prefix) - ✅ AP/CA regions supported (ap./ca. prefixes) - ✅ ON_DEMAND models unaffected - ✅ Backwards compatible with existing configurations ### Breaking Changes None. This change is fully backwards compatible. ## Type of change - [ ] Bug fix - [X] New feature - [ ] Breaking change - [ ] Documentation - [ ] Performance improvement ## Scope check - [X] I read the [Contributing Guide](https://github.com/crmne/ruby_llm/blob/main/CONTRIBUTING.md) - [X] This aligns with RubyLLM's focus on **LLM communication** - [X] This isn't application-specific logic that belongs in user code - [X] This benefits most users, not just my specific use case ## Quality check - [X] I ran `overcommit --install` and all hooks pass - [X] I tested my changes thoroughly - [X] I updated documentation if needed - [X] I didn't modify auto-generated files manually (`models.json`, `aliases.json`) ## API changes - [ ] Breaking change - [ ] New public methods/classes - [ ] Changed method signatures - [ ] No API changes --------- Co-authored-by: Enrique <enrique.rolon@qonto.com> Co-authored-by: Carmine Paolino <carmine@paolino.me>
diff --git a/lib/ruby_llm/providers/bedrock/models.rb b/lib/ruby_llm/providers/bedrock/models.rb
@@ -72,7 +72,25 @@ def model_id_with_region(model_id, model_data)
           return model_id unless model_data['inferenceTypesSupported']&.include?('INFERENCE_PROFILE')
           return model_id if model_data['inferenceTypesSupported']&.include?('ON_DEMAND')
 
-          "us.#{model_id}"
+          desired_region_prefix = inference_profile_region_prefix
+
+          # Return unchanged if model already has the correct region prefix
+          return model_id if model_id.start_with?("#{desired_region_prefix}.")
+
+          # Remove any existing region prefix (e.g., "us.", "eu.", "ap.")
+          clean_model_id = model_id.sub(/^[a-z]{2}\./, '')
+
+          # Apply the desired region prefix
+          "#{desired_region_prefix}.#{clean_model_id}"
+        end
+
+        def inference_profile_region_prefix
+          # Extract region prefix from bedrock_region (e.g., "eu-west-3" -> "eu")
+          region = @config.bedrock_region.to_s
+          return 'us' if region.empty? # Default fallback
+
+          # Take first two characters as the region prefix
+          region[0, 2]
         end
       end
     end
diff --git a/spec/ruby_llm/providers/bedrock/models_spec.rb b/spec/ruby_llm/providers/bedrock/models_spec.rb
@@ -37,7 +37,12 @@
       end
 
       it 'adds us. prefix to model ID' do
-        model_info = described_class.create_model_info(model_data, slug, capabilities)
+        # Mock a provider instance to test the region functionality
+        allow(RubyLLM.config).to receive(:bedrock_region).and_return('us-east-1')
+        provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+        provider.extend(described_class)
+
+        model_info = provider.send(:create_model_info, model_data, slug, capabilities)
         expect(model_info.id).to eq('us.anthropic.claude-3-7-sonnet-20250219-v1:0')
       end
     end
@@ -57,7 +62,12 @@
       end
 
       it 'does not add us. prefix to model ID' do
-        model_info = described_class.create_model_info(model_data, slug, capabilities)
+        # Mock a provider instance to test the region functionality
+        allow(RubyLLM.config).to receive(:bedrock_region).and_return('us-east-1')
+        provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+        provider.extend(described_class)
+
+        model_info = provider.send(:create_model_info, model_data, slug, capabilities)
         expect(model_info.id).to eq('anthropic.claude-3-5-sonnet-20240620-v1:0')
       end
     end
@@ -77,7 +87,12 @@
       end
 
       it 'does not add us. prefix to model ID' do
-        model_info = described_class.create_model_info(model_data, slug, capabilities)
+        # Mock a provider instance to test the region functionality
+        allow(RubyLLM.config).to receive(:bedrock_region).and_return('us-east-1')
+        provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+        provider.extend(described_class)
+
+        model_info = provider.send(:create_model_info, model_data, slug, capabilities)
         expect(model_info.id).to eq('anthropic.claude-3-5-sonnet-20240620-v1:0')
       end
     end
@@ -96,9 +111,118 @@
       end
 
       it 'does not add us. prefix to model ID' do
-        model_info = described_class.create_model_info(model_data, slug, capabilities)
+        # Mock a provider instance to test the region functionality
+        allow(RubyLLM.config).to receive(:bedrock_region).and_return('us-east-1')
+        provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+        provider.extend(described_class)
+
+        model_info = provider.send(:create_model_info, model_data, slug, capabilities)
         expect(model_info.id).to eq('anthropic.claude-3-5-sonnet-20240620-v1:0')
       end
     end
   end
+
+  # New specs for region-aware inference profile handling
+  describe '#model_id_with_region with region awareness' do
+    let(:provider_instance) do
+      allow(RubyLLM.config).to receive(:bedrock_region).and_return('eu-west-3')
+      provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+      provider.extend(described_class)
+      provider
+    end
+
+    context 'with EU region configured' do
+      let(:inference_profile_model) do
+        {
+          'modelId' => 'anthropic.claude-3-7-sonnet-20250219-v1:0',
+          'inferenceTypesSupported' => ['INFERENCE_PROFILE']
+        }
+      end
+
+      let(:us_prefixed_model) do
+        {
+          'modelId' => 'us.anthropic.claude-opus-4-1-20250805-v1:0',
+          'inferenceTypesSupported' => ['INFERENCE_PROFILE']
+        }
+      end
+
+      it 'adds eu. prefix for inference profile models' do
+        result = provider_instance.send(:model_id_with_region,
+                                        inference_profile_model['modelId'],
+                                        inference_profile_model)
+        expect(result).to eq('eu.anthropic.claude-3-7-sonnet-20250219-v1:0')
+      end
+
+      it 'adds eu. prefix to us. prefixed model' do
+        result = provider_instance.send(:model_id_with_region,
+                                        us_prefixed_model['modelId'],
+                                        us_prefixed_model)
+        expect(result).to eq('eu.anthropic.claude-opus-4-1-20250805-v1:0')
+      end
+    end
+
+    context 'with AP region configured' do
+      let(:provider_instance) do
+        allow(RubyLLM.config).to receive(:bedrock_region).and_return('ap-south-1')
+        provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+        provider.extend(described_class)
+        provider
+      end
+
+      it 'adds ap. prefix to existing us. prefixed model' do
+        model_data = {
+          'modelId' => 'us.anthropic.claude-opus-4-1-20250805-v1:0',
+          'inferenceTypesSupported' => ['INFERENCE_PROFILE']
+        }
+
+        result = provider_instance.send(:model_id_with_region,
+                                        model_data['modelId'],
+                                        model_data)
+        expect(result).to eq('ap.anthropic.claude-opus-4-1-20250805-v1:0')
+      end
+    end
+
+    context 'with region prefix edge cases' do
+      it 'handles empty region gracefully' do
+        allow(RubyLLM.config).to receive(:bedrock_region).and_return('')
+        provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+        provider.extend(described_class)
+
+        model_data = {
+          'modelId' => 'anthropic.claude-opus-4-1-20250805-v1:0',
+          'inferenceTypesSupported' => ['INFERENCE_PROFILE']
+        }
+
+        result = provider.send(:model_id_with_region,
+                               model_data['modelId'],
+                               model_data)
+        expect(result).to eq('us.anthropic.claude-opus-4-1-20250805-v1:0')
+      end
+
+      it 'extracts region prefix from various AWS regions' do
+        regions_and_expected_prefixes = {
+          'eu-west-3' => 'eu',
+          'ap-south-1' => 'ap',
+          'ca-central-1' => 'ca',
+          'sa-east-1' => 'sa'
+        }
+
+        regions_and_expected_prefixes.each do |region, expected_prefix|
+          allow(RubyLLM.config).to receive(:bedrock_region).and_return(region)
+          provider = RubyLLM::Providers::Bedrock.new(RubyLLM.config)
+          provider.extend(described_class)
+
+          model_data = {
+            'modelId' => 'anthropic.claude-opus-4-1-20250805-v1:0',
+            'inferenceTypesSupported' => ['INFERENCE_PROFILE']
+          }
+
+          result = provider.send(:model_id_with_region,
+                                 model_data['modelId'],
+                                 model_data)
+          expect(result).to eq("#{expected_prefix}.anthropic.claude-opus-4-1-20250805-v1:0")
+        end
+      end
+    end
+  end
 end
