Add proxy-bypass option to prevent using proxy depending on rule #2187

AlexKamaev · 2018-03-02T15:05:25Z

No description provided.

testcafe-build-bot · 2018-03-02T16:48:15Z

❌ Tests for the commit 86e06b0 have failed. See details:

testcafe-build-bot · 2018-03-03T08:45:18Z

❌ Tests for the commit 4db9ede have failed. See details:

testcafe-build-bot · 2018-03-03T09:34:09Z

✅ Tests for the commit c34ef94 have passed. See details:

…Express#2187)

AlexKamaev · 2018-03-05T06:34:59Z

see API in tests

helen-dikareva · 2018-03-05T07:03:32Z

test/server/match-url-test.js

+var expect   = require('chai').expect;
+var matchUrl = require('../../lib/utils/check-url');
+
+it('Should fallback to the default reporter if reporter was not set', function () {


Strange name for the test

AlexKamaev · 2018-03-05T09:28:45Z

@testcafe-build-bot \retest

testcafe-build-bot · 2018-03-05T10:15:25Z

❌ Tests for the commit 7376ad1 have failed. See details:

AndreyBelym · 2018-03-05T09:01:50Z

src/cli/argument-parser.js

            .option('--ports <port1,port2>', 'specify custom port numbers')
            .option('--hostname <name>', 'specify the hostname')
            .option('--proxy <host>', 'specify the host of the proxy server')
+            .option('--proxy-bypass <url1,url2>', 'specify the urls to bypass the proxy server')


I think it must be more like the reporters method description: <host-wildcard[,...]>

\cc @VasilyStrelyaev

AndreyBelym · 2018-03-05T10:26:33Z

test/server/match-url-test.js

+    expect(matchUrl('127.0.0.1', rule)).to.be.true;
+    expect(matchUrl('http://127.0.0.1', rule)).to.be.true;
+    expect(matchUrl('https://127.0.0.1', rule)).to.be.true;
+    expect(matchUrl('https://127.0.0.1', rule)).to.be.true;


yes. mistake

helen-dikareva · 2018-03-05T08:54:11Z

test/server/match-url-test.js

+    expect(matchUrl('127.0.0.1', rule)).to.be.true;
+    expect(matchUrl('http://127.0.0.1', rule)).to.be.true;
+    expect(matchUrl('https://127.0.0.1', rule)).to.be.true;
+    expect(matchUrl('https://127.0.0.1', rule)).to.be.true;


This is the same line as above

helen-dikareva · 2018-03-05T09:50:07Z

src/utils/check-url.js

+const containsWildcardPlaceholder = new RegExp(wildcardPlaceholder, 'g');
+
+function parseUrl (url) {
+


we don't need this blank line

helen-dikareva · 2018-03-05T09:59:26Z

test/server/match-url-test.js

+var matchUrl = require('../../lib/utils/check-url');
+
+it('Should check does url match rule', function () {
+    var rule = ['google.com'];


We can use just string here, can't we?

helen-dikareva · 2018-03-05T10:21:42Z

src/cli/argument-parser.js

            .option('--ports <port1,port2>', 'specify custom port numbers')
            .option('--hostname <name>', 'specify the hostname')
            .option('--proxy <host>', 'specify the host of the proxy server')
+            .option('--proxy-bypass <url1,url2>', 'specify the urls to bypass the proxy server')


We need to check type of url template value and create error for a case when we cannot parse it

I thought it is not required to create error. If rule is incorrent test will run via proxy, and that's all. Rule just will be ignored. Don't you agree?

testcafe-build-bot · 2018-03-05T11:07:03Z

✅ Tests for the commit 7376ad1 have passed. See details:

VasilyStrelyaev · 2018-03-05T14:11:57Z

src/cli/argument-parser.js

            .option('--ports <port1,port2>', 'specify custom port numbers')
            .option('--hostname <name>', 'specify the hostname')
            .option('--proxy <host>', 'specify the host of the proxy server')
+            .option('--proxy-bypass <url1,url2>', 'specify the urls to bypass the proxy server')


.option('--proxy-bypass <rules>', 'specify a comma-separated list of rules that define URLs accessed bypassing the proxy server')

Regarding #2187 (comment) we can just add Rules consist of URLs with wildcards enabled.
Is that simpler. @AndreyBelym ?

testcafe-build-bot · 2018-03-05T15:31:59Z

❌ Tests for the commit b7736b5 have failed. See details:

testcafe-build-bot · 2018-03-06T09:55:15Z

❌ Tests for the commit b054111 have failed. See details:

AlexKamaev · 2018-03-06T09:55:52Z

@testcafe-build-bot \retest

testcafe-build-bot · 2018-03-06T10:21:49Z

❌ Tests for the commit b054111 have failed. See details:

testcafe-build-bot · 2018-03-06T11:16:15Z

✅ Tests for the commit b054111 have passed. See details:

AlexKamaev · 2018-03-06T11:19:56Z

FPR

testcafe-build-bot · 2018-03-06T13:33:23Z

✅ Tests for the commit 06a730f have passed. See details:

n0v1 · 2018-03-07T14:13:49Z

Great to see this feature beeing added! Thank you!

Is this expected to work for ressources like CSS stylesheets or JavaScript files that are linked from the page under test?

I'm sitting behind a corporate proxy and want to test a page that is served by a web server on localhost. The page loads a JavaScript library from a remote host. So the request to load the page should bypass the proxy, the request to get the library should go through the proxy.

I call TestCafé CLI with --proxy 172.30.109.30:8080 --proxy-bypass 127.0.0.1:8000. The page and all local ressources load correctly but the remote script does not. It keeps loading forever until a timeout is hit eventually.

To test in isolation I created this small repo: n0v1/testcafe-proxy-bypass-test

Update:

When debugging this, testcafe-hammerhead/src/request-pipeline/destination-request/_onError receives this error object:

{
  Error: connect ETIMEDOUT 104.16.86.20:80
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1182:14)
  code: 'ETIMEDOUT',
  errno: 'ETIMEDOUT',
  syscall: 'connect',
  address: '104.16.86.20',
  port: 80
}

this.opts = ... (click to expand)

this.opts = {
  url: 'http://cdn.jsdelivr.net/npm/[email protected]/lodash.min.js',
  protocol: 'http:',
  hostname: 'cdn.jsdelivr.net',
  host: 'cdn.jsdelivr.net',
  port: undefined,
  path: '/npm/[email protected]/lodash.min.js',
  method: 'GET',
  credentials: null,
  body: <Buffer >,
  isXhr: false,
  proxy: null,
  headers: {
    host: 'cdn.jsdelivr.net',
    'user-agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0',
    accept: '*/*',
    'accept-language': 'de,en-US;q=0.7,en;q=0.3',
    'accept-encoding': 'gzip, deflate',
    referer: 'http://127.0.0.1:8000/test.html',
    connection: 'keep-alive'
  },
  agent: {
    domain: null,
    _events: { free: [Function] },
    _eventsCount: 1,
    _maxListeners: undefined,
    defaultPort: 80,
    protocol: 'http:',
    options: { secureProtocol: undefined, keepAlive: true, path: null },
    requests: {},
    sockets: { 'cdn.jsdelivr.net:80::': [Array] },
    freeSockets: {},
    keepAliveMsecs: 1000,
    keepAlive: true,
    maxSockets: Infinity,
    maxFreeSockets: 256
  } 
}

AlexKamaev · 2018-03-12T14:06:27Z

Hi
Now this feature is implemented only to bypass proxy for specific tests according to the test page url. But your case seems to be important, so I think we'll support it as well.

amirse80 · 2018-09-06T15:16:47Z

Leave a comment

amirse80 · 2018-09-06T15:21:42Z

Hello, can you please clarify how to use this bypass?
E.g: If I run my simple test case from CMD "TestCafe>testcafe chrome test1.js" and in the testfile I have an URL which is copmany internal how do I start it and make sure that URL is bypassed?

The problem we have is that we first do a GET to one webpage but then during the load it is redirected to another one. And in TestCafe the first page loads correctly but the next one gets UNAUTHORIZED. However pasting that second URL in the same browser manually loads the page correctly.

So, what can I do in TestCafe to also bypass proxy for the redirected page?

AlexKamaev · 2018-09-06T15:55:18Z

Hello @amirse80
TestCafe CLI supports two options for using proxy. Please refer to the following articles:
--proxy and --proxy-bypass.
According to our documentation, the proxyoption specifies the proxy server used in your local network to access the Internet. If you do not have access to the second URL, I suppose that you need to use the proxy option.
The proxy-bypass option can be used together with the proxy option. It's required when you use proxy, but don't need to use proxy only for some specific resources.

amirse80 · 2018-09-07T07:50:19Z

Hello,
Thank you for the input. Unfortunatelly it didn´t solve my problem.
The issue is like this.

I navigate to an company internal page. (.page `http://dap-internal.company.page´)
The chrome is started and initially navigates to the correct server.
During the page opening the URL changes, redirects, to http://pen-internal.company.page. Now this new URL works fine in the same browser that testcafe has started IF i copy the real URL manually into the new tab BUT testcafe adds http://172.30.48.47:61019/DSc6im7OL/ to the URL which makes it not working. So instead of the real redirected URL http://pen-internal.company.page the redirected URL is http://172.30.48.47:61019/DSc6im7OL/http://pen.internal.company.page!

So if I would be able to just remove this part that testCafe adds, http://172.30.48.47:61019/DSc6im7OL/, it would probably work, because it does work if I manually add the redirected URL in the same browser.

…Express#2187)

AlexKamaev force-pushed the proxy-bypass branch from 86e06b0 to 4db9ede Compare March 3, 2018 08:15

AlexKamaev force-pushed the proxy-bypass branch from 4db9ede to c34ef94 Compare March 3, 2018 09:04

AlexKamaev changed the title ~~[WIP]noproxy~~ Add proxy-bypass option to prevent using proxy depending on rule Mar 5, 2018

AlexKamaev requested review from AndreyBelym and helen-dikareva March 5, 2018 06:31

Add proxy-bypass option to prevent using proxy depending on rule (Dev…

447cb15

…Express#2187)

AlexKamaev force-pushed the proxy-bypass branch from c34ef94 to 447cb15 Compare March 5, 2018 06:33

AlexKamaev requested a review from VasilyStrelyaev March 5, 2018 06:33

remove extra space

def3e0f

helen-dikareva reviewed Mar 5, 2018

View reviewed changes

fix test name

7376ad1

AlexKamaev closed this Mar 5, 2018

AlexKamaev reopened this Mar 5, 2018

AndreyBelym suggested changes Mar 5, 2018

View reviewed changes

helen-dikareva requested changes Mar 5, 2018

View reviewed changes

VasilyStrelyaev requested changes Mar 5, 2018

View reviewed changes

add some additional validations

b5a7efa

AlexKamaev force-pushed the proxy-bypass branch from 920f477 to b5a7efa Compare March 5, 2018 14:33

fix arg description

b7736b5

AlexKamaev force-pushed the proxy-bypass branch from 299285b to 7079298 Compare March 6, 2018 08:38

throw error in proxy-bypass is not string and refactor runner validation

b054111

AlexKamaev force-pushed the proxy-bypass branch from 7079298 to b054111 Compare March 6, 2018 08:52

VasilyStrelyaev approved these changes Mar 6, 2018

View reviewed changes

helen-dikareva approved these changes Mar 6, 2018

View reviewed changes

fix some formatting mistakes

06a730f

AlexKamaev force-pushed the proxy-bypass branch from b60c2de to 06a730f Compare March 6, 2018 12:44

AndreyBelym approved these changes Mar 6, 2018

View reviewed changes

AndreyBelym merged commit ff555e3 into DevExpress:master Mar 6, 2018

AlexKamaev mentioned this pull request Mar 6, 2018

Need for proxy ignore/bypass feature #1791

Closed

kirovboris pushed a commit to kirovboris/testcafe-phoenix that referenced this pull request Dec 18, 2019

Add proxy-bypass option to prevent using proxy depending on rule (Dev…

37145f7

…Express#2187)

		const containsWildcardPlaceholder = new RegExp(wildcardPlaceholder, 'g');

		function parseUrl (url) {

Add proxy-bypass option to prevent using proxy depending on rule #2187

Add proxy-bypass option to prevent using proxy depending on rule #2187

Uh oh!

Conversation

AlexKamaev commented Mar 2, 2018

Uh oh!

testcafe-build-bot commented Mar 2, 2018

Uh oh!

testcafe-build-bot commented Mar 3, 2018

Uh oh!

testcafe-build-bot commented Mar 3, 2018

Uh oh!

AlexKamaev commented Mar 5, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexKamaev commented Mar 5, 2018

Uh oh!

testcafe-build-bot commented Mar 5, 2018

Uh oh!

AndreyBelym Mar 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

testcafe-build-bot commented Mar 5, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

testcafe-build-bot commented Mar 5, 2018

Uh oh!

testcafe-build-bot commented Mar 6, 2018

Uh oh!

AlexKamaev commented Mar 6, 2018

Uh oh!

testcafe-build-bot commented Mar 6, 2018

Uh oh!

testcafe-build-bot commented Mar 6, 2018

Uh oh!

AlexKamaev commented Mar 6, 2018

Uh oh!

testcafe-build-bot commented Mar 6, 2018

Uh oh!

n0v1 commented Mar 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexKamaev commented Mar 12, 2018

Uh oh!

amirse80 commented Sep 6, 2018

Uh oh!

amirse80 commented Sep 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexKamaev commented Sep 6, 2018

Uh oh!

amirse80 commented Sep 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

AndreyBelym Mar 5, 2018 •

edited

Loading

n0v1 commented Mar 7, 2018 •

edited

Loading

amirse80 commented Sep 6, 2018 •

edited

Loading

amirse80 commented Sep 7, 2018 •

edited

Loading