Fix Failing Build Tests on Windows & MacOS

[emmacasolin]

Specification

This is not about integration testing, this is about our unit tests

Not all of our tests are passing on Windows/Mac, both in the CI/CD and on matrix-win-1/matrix-mac-1. Some of these are due to obvious platform differences (e.g. the usage of / vs \), but others may be more difficult to debug.

Failing tests on windows:

• tests/bin
  • /vaults/vaults.test.ts
    • should clone and pull a vault
  • /agent
    • /start.test.ts
      • start in foreground
      • start in background
      • concurrent starts results in 1 success
      • concurrent with bootstrap results in 1 success
      • start with existing state
      • start when interrupted, requires fresh on next start
      • start from recovery code
      • start with network configuration
    • /stop.test.ts
      • stopping is idempotent during concurrent calls and STOPPING or DEAD status
      • stopping starting agent results in error
    • /status.test.ts
      • status on STARTING, STOPPING, DEAD agent
  • /secrets/secrets.test.ts
    • commandNewDir › should make a directory
    • commandNewDirSecret › should add a directory of secrets
  • /bootstrap.test.ts
    • concurrent bootstrapping results in 1 success
    • bootstrap when interrupted, requires fresh on next bootstrap
  • /utils.test.ts
    • errors in human and json format
• tests/keys/KeyManager.test.ts
  • can reset root key pair
  • can renew root key pair
• tests/network/Proxy.test.ts
  • connection to port 0 fails
  • open connection to port 0 fails
  • closed connection due to ending server
• tests/nodes/NodeConnection.test.ts
  • should call `killSelf and throw if the server exit's during testUnaryFail
  • should call `killSelf and throw if the server kill's during testUnaryFail
  • should call `killSelf and throw if the server sigkill's during testUnaryFail
  • should call `killSelf and throw if the server exit's during testStreamFail
  • should call `killSelf and throw if the server kill's during testStreamFail
  • should call `killSelf and throw if the server sigkill's during testStreamFail
• tests/vaults
  • /VaultInternal.test.ts
    • does not allow changing to an unrecognised commit
    • cannot checkout old commits after branching commit
    • can recover from dirty state
    • clean errant commits recovering from dirty state
    • garbage collection
    • can create an existing vault with CreateVaultInternal
  • /VaultManager.test.ts
    • With remote agents › clone vaults from a remote keynode using a vault name
    • With remote agents › clone vaults from a remote keynode using a vault name with no history
    • With remote agents › clone and pull vaults using a vault id
    • With remote agents › should reject Pulling when permissions are not set
    • With remote agents › can pull a cloned vault
    • With remote agents › manage pulling from different remotes
    • With remote agents › throw when trying to commit to a cloned vault
    • With remote agents › pullVault respects locking
  • /utils.test.ts
    • EFS can be read recursively
  • /VaultOps.test.ts
    • able to make directories
    • deleting a secret
    • deleting a secret within a directory
    • renaming a secret within a directory
    • listing secrets
    • adding hidden files and directories
    • updating and deleting hidden files and directories
    • adding a directory of 1 secret
    • adding a directory of 100 secrets with some secrets already existing

Failing tests on Mac (all timeouts, even when only a single test is run):

• tests/network/proxy.test.ts
  • connection to port 0 fails
  • open connection to port 0 fails
• tests/bin/agent/start.test.ts
  • start with network configuration

Additional context

Tasks

1. Group tests by similar failures
2. ...
3. ...

[emmacasolin]

Full output from failing tests:

Summary of all failing tests
 FAIL  tests/bin/vaults/vaults.test.ts (90.027 s)
  ● CLI vaults › should clone and pull a vault

    expect(received).toBe(expected) // Object.is equality

    Expected: 0
    Received: 255

      268 |
      269 |       let result = await testBinUtils.pkStdio([...command], {}, dataDir);
    > 270 |       expect(result.exitCode).toBe(0);
          |                               ^
      271 |
      272 |       const clonedVaultId = await polykeyAgent.vaultManager.getVaultId(
      273 |         vaultName,

      at Object.<anonymous> (tests/bin/vaults/vaults.test.ts:270:31)

 FAIL  tests/bin/agent/start.test.ts (170.883 s)
  ● start › start in foreground

    spawn ts-node ENOENT



  ● start › start in foreground

    thrown: undefined

      28 |     });
      29 |   });
    > 30 |   test(
         |   ^
      31 |     'start in foreground',
      32 |     async () => {
      33 |       const password = 'abc123';

      at tests/bin/agent/start.test.ts:30:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

  ● start › start in background

    spawn ts-node ENOENT



  ● start › start in background

    thrown: undefined

       99 |     global.defaultTimeout * 2,
      100 |   );
    > 101 |   test(
          |   ^
      102 |     'start in background',
      103 |     async () => {
      104 |       const password = 'abc123';

      at tests/bin/agent/start.test.ts:101:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

  ● start › concurrent starts results in 1 success

    spawn ts-node ENOENT



  ● start › concurrent starts results in 1 success

    spawn ts-node ENOENT



  ● start › concurrent starts results in 1 success

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      200 |     global.defaultTimeout * 2,
      201 |   );
    > 202 |   test(
          |   ^
      203 |     'concurrent starts results in 1 success',
      204 |     async () => {
      205 |       const password = 'abc123';

      at tests/bin/agent/start.test.ts:202:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

  ● start › concurrent with bootstrap results in 1 success

    spawn ts-node ENOENT



  ● start › concurrent with bootstrap results in 1 success

    spawn ts-node ENOENT



  ● start › concurrent with bootstrap results in 1 success

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      298 |     global.defaultTimeout * 2,
      299 |   );
    > 300 |   test(
          |   ^
      301 |     'concurrent with bootstrap results in 1 success',
      302 |     async () => {
      303 |       const password = 'abc123';

      at tests/bin/agent/start.test.ts:300:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

  ● start › start with existing state

    spawn ts-node ENOENT



  ● start › start with existing state

    thrown: undefined

      390 |     global.defaultTimeout * 2,
      391 |   );
    > 392 |   test(
          |   ^
      393 |     'start with existing state',
      394 |     async () => {
      395 |       const password = 'abc123';

      at tests/bin/agent/start.test.ts:392:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

  ● start › start when interrupted, requires fresh on next start

    spawn ts-node ENOENT



  ● start › start when interrupted, requires fresh on next start

    thrown: undefined

      470 |     global.defaultTimeout * 2,
      471 |   );
    > 472 |   test(
          |   ^
      473 |     'start when interrupted, requires fresh on next start',
      474 |     async () => {
      475 |       const password = 'password';

      at tests/bin/agent/start.test.ts:472:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

  ● start › start from recovery code

    spawn ts-node ENOENT



  ● start › start from recovery code

    thrown: undefined

      580 |     global.defaultTimeout * 2,
      581 |   );
    > 582 |   test(
          |   ^
      583 |     'start from recovery code',
      584 |     async () => {
      585 |       const password1 = 'abc123';

      at tests/bin/agent/start.test.ts:582:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

  ● start › start with network configuration

    spawn ts-node ENOENT



  ● start › start with network configuration

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      714 |     global.defaultTimeout * 3,
      715 |   );
    > 716 |   test(
          |   ^
      717 |     'start with network configuration',
      718 |     async () => {
      719 |       const status = new Status({

      at tests/bin/agent/start.test.ts:716:3
      at Object.<anonymous> (tests/bin/agent/start.test.ts:16:1)

 FAIL  tests/bin/secrets/secrets.test.ts (41.16 s)
  ● CLI secrets › commandNewDir › should make a directory

    ErrorEncryptedFSError: ENOENT: no such file or directory, dir1\MySecret1

      67 |   for (const dirent of dirents) {
      68 |     const res = path.join(dir, dirent.toString());
    > 69 |     const stat = await fs.promises.stat(res);
         |                  ^
      70 |     if (stat.isDirectory()) {
      71 |       yield* readdirRecursively(fs, res);
      72 |     } else if (stat.isFile()) {

      at node_modules/encryptedfs/src/EncryptedFS.ts:2932:15
      at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12)
      at readdirRecursively (src/vaults/utils.ts:69:18)
      at Object.readdirRecursively (src/vaults/utils.ts:71:7)
      at src/vaults/VaultOps.ts:256:22
      at src/vaults/VaultInternal.ts:427:14
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at Object.listSecrets (src/vaults/VaultOps.ts:254:10)
      at tests/bin/secrets/secrets.test.ts:169:22
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)

  ● CLI secrets › commandNewDirSecret › should add a directory of secrets

    expect(received).toStrictEqual(expected) // deep equality

    - Expected  - 3
    + Received  + 3

      Array [
    -   "secrets/secret-1",
    -   "secrets/secret-2",
    -   "secrets/secret-3",
    +   "secrets\\secret-1",
    +   "secrets\\secret-2",
    +   "secrets\\secret-3",
      ]

      269 |       await polykeyAgent.vaultManager.withVaults([vaultId], async (vault) => {
      270 |         const list = await vaultOps.listSecrets(vault);
    > 271 |         expect(list.sort()).toStrictEqual([
          |                             ^
      272 |           'secrets/secret-1',
      273 |           'secrets/secret-2',
      274 |           'secrets/secret-3',

      at tests/bin/secrets/secrets.test.ts:271:29
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withVaults (src/vaults/VaultManager.ts:971:12)
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at Object.<anonymous> (tests/bin/secrets/secrets.test.ts:269:7)

 FAIL  tests/bin/bootstrap.test.ts (54.965 s)
  ● bootstrap › concurrent bootstrapping results in 1 success

    spawn ts-node ENOENT



  ● bootstrap › concurrent bootstrapping results in 1 success

    spawn ts-node ENOENT



  ● bootstrap › concurrent bootstrapping results in 1 success

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      105 |     global.defaultTimeout * 2,
      106 |   );
    > 107 |   test(
          |   ^
      108 |     'concurrent bootstrapping results in 1 success',
      109 |     async () => {
      110 |       const password = 'password';

      at tests/bin/bootstrap.test.ts:107:3
      at Object.<anonymous> (tests/bin/bootstrap.test.ts:10:1)
          at runMicrotasks (<anonymous>)

  ● bootstrap › bootstrap when interrupted, requires fresh on next bootstrap

    spawn ts-node ENOENT



  ● bootstrap › bootstrap when interrupted, requires fresh on next bootstrap

    thrown: undefined

      186 |     global.defaultTimeout * 2,
      187 |   );
    > 188 |   test(
          |   ^
      189 |     'bootstrap when interrupted, requires fresh on next bootstrap',
      190 |     async () => {
      191 |       const password = 'password';

      at tests/bin/bootstrap.test.ts:188:3
      at Object.<anonymous> (tests/bin/bootstrap.test.ts:10:1)
          at runMicrotasks (<anonymous>)

 FAIL  tests/bin/agent/stop.test.ts (70.078 s)
  ● stop › stopping is idempotent during concurrent calls and STOPPING or DEAD status

    expect(received).toBe(expected) // Object.is equality

    Expected: 0
    Received: "ENOENT"

      152 |         expect(agentStop1.exitCode).toBe(0);
      153 |       } else {
    > 154 |         expect(agentStop1.exitCode).toBe(0);
          |                                     ^
      155 |         expect(agentStop2.exitCode).toBe(0);
      156 |       }
      157 |       expect(agentStop3.exitCode).toBe(0);

      at Object.<anonymous> (tests/bin/agent/stop.test.ts:154:37)

  ● stop › stopping starting agent results in error

    spawn ts-node ENOENT



  ● stop › stopping starting agent results in error

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      160 |     global.defaultTimeout * 2,
      161 |   );
    > 162 |   test(
          |   ^
      163 |     'stopping starting agent results in error',
      164 |     async () => {
      165 |       const password = 'abc123';

      at tests/bin/agent/stop.test.ts:162:3
      at Object.<anonymous> (tests/bin/agent/stop.test.ts:12:1)

 FAIL  tests/bin/utils.test.ts
  ● bin/utils › errors in human and json format

    expect(received).toBe(expected) // Object.is equality

    - Expected  - 1
    + Received  + 1

    - {"type":"TypeError","data":{"message":"some error","stack":"TypeError: some error\n    at Object.<anonymous> (C:\Users\Emma Casolin\Projects\js-polykey\tests\bin\utils.test.ts:84:27)\n    at Promise.then.completed (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\utils.js:333:28)\n    at new Promise (<anonymous>)\n    at callAsyncCircusFn (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\utils.js:259:10)\n    at _callCircusTest (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:277:40)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at _runTest (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:209:3)\n    at _runTestsForDescribeBlock (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:97:9)\n    at _runTestsForDescribeBlock (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:91:9)\n    at run (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:31:3)"}}
    + {"type":"TypeError","data":{"message":"some error","stack":"TypeError: some error\n    at Object.<anonymous> (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\tests\\bin\\utils.test.ts:84:27)\n    at Promise.then.completed (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\utils.js:333:28)\n    at new Promise (<anonymous>)\n    at callAsyncCircusFn (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\utils.js:259:10)\n    at _callCircusTest (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:277:40)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at _runTest (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:209:3)\n    at _runTestsForDescribeBlock (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:97:9)\n    at _runTestsForDescribeBlock (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:91:9)\n    at run (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:31:3)"}}
      ↵

      170 |     expect(
      171 |       binUtils.outputFormatter({ type: 'json', data: standardError }),
    > 172 |     ).toBe(
          |       ^
      173 |       `{"type":"${standardError.name}","data":{"message":"${
      174 |         standardError.message
      175 |       }","stack":"${standardError.stack?.replaceAll('\n', '\\n')}"}}\n`,

      at Object.<anonymous> (tests/bin/utils.test.ts:172:7)

 FAIL  tests/bin/agent/status.test.ts (44.001 s)
  ● status › status on STARTING, STOPPING, DEAD agent

    spawn ts-node ENOENT



  ● status › status on STARTING, STOPPING, DEAD agent

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      25 |     });
      26 |   });
    > 27 |   test(
         |   ^
      28 |     'status on STARTING, STOPPING, DEAD agent',
      29 |     async () => {
      30 |       // This test must create its own agent process

      at tests/bin/agent/status.test.ts:27:3
      at Object.<anonymous> (tests/bin/agent/status.test.ts:11:1)
          at runMicrotasks (<anonymous>)


Test Suites: 7 failed, 25 passed, 32 total
Tests:       17 failed, 1 todo, 95 passed, 113 total
Snapshots:   0 total
Time:        362.339 s
Ran all test suites matching /tests\\bin/i.
GLOBAL TEARDOWN
Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-niloUA

> jest "tests/keys/keymanager" "--maxWorkers=50%"

Determining test suites to run...
GLOBAL SETUP
Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-BuA71v
 FAIL  tests/keys/KeyManager.test.ts (63.31 s)
  KeyManager
    √ KeyManager readiness (920 ms)
    √ constructs root key pair, root cert, root certs and db key (898 ms)
    √ creates a recovery code and can recover from the same code (26689 ms)
    √ create deterministic keypair with recovery code (9273 ms)
    √ uses WorkerManager for generating root key pair (478 ms)
    √ encrypting and decrypting with root key (491 ms)
    √ uses WorkerManager for encryption and decryption with root key (575 ms)
    √ encrypting beyond maximum size (467 ms)
    √ signing and verifying with root key (509 ms)
    √ uses WorkerManager for signing and verifying with root key (523 ms)
    √ can change root key password (1860 ms)
    √ can reset root certificate (2491 ms)
    × can reset root key pair (3008 ms)
    × can renew root key pair (3030 ms)
    √ order of certificate chain should be leaf to root (5422 ms)
    dbKey
      √ Creates a key when started. (509 ms)
      √ Throws an exception when it fails to parse the key. (913 ms)
      √ key remains unchanged when resetting keys. (1346 ms)
      √ key remains unchanged when renewing keys. (1350 ms)

  ● KeyManager › can reset root key pair

    EBUSY: resource busy or locked, unlink 'C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-8sVU03\db\000005.ldb'



  ● KeyManager › can renew root key pair

    EBUSY: resource busy or locked, unlink 'C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-dJ4AxX\db\000005.ldb'



Test Suites: 1 failed, 1 total
Tests:       2 failed, 17 passed, 19 total
Snapshots:   0 total
Time:        63.548 s
Ran all test suites matching /tests\\keys\\keymanager/i.

PS C:\Users\Emma Casolin\Projects\js-polykey> npm run test tests/network/proxy -- --maxWorkers=50%

> @matrixai/polykey@1.0.0 test
> jest "tests/network/proxy" "--maxWorkers=50%"

Determining test suites to run...
GLOBAL SETUP
Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-WkqxaW
ERROR:Proxy CONNECT bad request:Failed CONNECT to 127.0.0.1:80 - ErrorProxyConnectAuth
ERROR:Proxy CONNECT bad request:Failed CONNECT to 0.0.0.0:80 - ErrorProxyConnectInvalidUrl
ERROR:Proxy CONNECT bad request:Failed CONNECT - ErrorProxyConnectMissingNodeId
ERROR:Proxy CONNECT bad request:Failed CONNECT - ErrorProxyConnectInvalidUrl
ERROR:Proxy connection timeout:Failed CONNECT to 127.0.0.1:61837 - ErrorConnectionStartTimeout
ERROR:Proxy connection reset:Failed CONNECT to 127.0.0.1:61839 - ErrorConnectionStart: UTP_ECONNRESET
ERROR:Proxy missing certificates:Failed CONNECT to 127.0.0.1:61846 - ErrorConnectionStart: 5604:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:c:\ws\deps\openssl\openssl\ssl\record\rec_layer_s3.c:1546:SSL alert number 40

ERROR:Proxy invalid node id:Failed CONNECT to 127.0.0.1:61850 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate
WARN:ConnectionForward 127.0.0.1:61864:Forward Error: ErrorConnectionTimeout
WARN:ConnectionForward 127.0.0.1:61866:Forward Error: ErrorConnectionTimeout
ERROR:Proxy port 0:Failed CONNECT to 127.0.0.1:0 - ErrorConnectionStartTimeout
WARN:ConnectionReverse 127.0.0.1:61886:Reverse Error: ErrorConnectionTimeout
WARN:Proxy test:Failed connection from 127.0.0.1:61886 - ErrorConnectionComposeTimeout
WARN:Proxy test:Failed connection from 127.0.0.1:61888 - ErrorCertChainEmpty: No certificates available to verify
WARN:ConnectionReverse 127.0.0.1:61888:Reverse Error: ErrorConnectionTimeout
WARN:Proxy test:Failed connection from 127.0.0.1:61892 - ErrorConnectionNotRunning
WARN:ConnectionReverse 127.0.0.1:61892:Reverse Error: ErrorConnectionEndTimeout
 FAIL  tests/network/Proxy.test.ts (75.495 s)
  Proxy
    √ proxy readiness (76 ms)
    √ HTTP CONNECT bad request failures to the forward proxy (46 ms)
    × connection to port 0 fails (20005 ms)
    √ connection start timeout due to hanging remote (4187 ms)
    √ connection reset due to ending remote (4604 ms)
    √ open connection fails due to missing certificates (31 ms)
    √ HTTP CONNECT fails due to missing certificates (26 ms)
    √ open connection fails due to invalid node id (209 ms)
    √ HTTP CONNECT fails due to invalid node id (66 ms)
    √ open connection success - forward initiates end (70 ms)
    √ open connection success - reverse initiates end (84 ms)
    √ HTTP CONNECT success - forward initiates end (80 ms)
    √ HTTP CONNECT success - reverse initiates end (174 ms)
    √ HTTP CONNECT success - client initiates end (233 ms)
    √ HTTP CONNECT success by opening connection first (84 ms)
    √ open connection keepalive timeout (1086 ms)
    √ HTTP CONNECT keepalive timeout (1178 ms)
    √ stopping the proxy with open forward connections (89 ms)
    √ open connection to multiple servers (90 ms)
    × open connection to port 0 fails (20007 ms)
    √ open connection timeout due to lack of ready signal (3016 ms)
    √ open connection success (8 ms)
    √ open connection to multiple clients (15 ms)
    × closed connection due to ending server (7 ms)
    √ connect timeout due to hanging client (3162 ms)
    √ connect fails due to missing client certificates (2126 ms)
    √ connect success (113 ms)
    √ stopping the proxy with open reverse connections (1083 ms)
    √ connectionEstablishedCallback is called when a ReverseConnection is established (59 ms)

  ● Proxy › connection to port 0 fails

    thrown: "Exceeded timeout of 20000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      242 |     await proxy.stop();
      243 |   });
    > 244 |   test('connection to port 0 fails', async () => {
          |   ^
      245 |     const proxy = new Proxy({
      246 |       authToken,
      247 |       logger: logger.getChild('Proxy port 0'),

      at tests/network/Proxy.test.ts:244:3
      at Object.<anonymous> (tests/network/Proxy.test.ts:108:1)

  ● Proxy › open connection to port 0 fails

    thrown: "Exceeded timeout of 20000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      2316 |     await proxy.stop();
      2317 |   });
    > 2318 |   test('open connection to port 0 fails', async () => {
           |   ^
      2319 |     const proxy = new Proxy({
      2320 |       logger: logger.getChild('Proxy port 0'),
      2321 |       authToken: '',

      at tests/network/Proxy.test.ts:2318:3
      at Object.<anonymous> (tests/network/Proxy.test.ts:108:1)

  ● Proxy › closed connection due to ending server

    expect(received).toBe(expected) // Object.is equality

    Expected: 1
    Received: 0

      2563 |     const utpSocketPort = utpSocket.address().port;
      2564 |     await proxy.openConnectionReverse(localHost, utpSocketPort as Port);
    > 2565 |     expect(proxy.getConnectionReverseCount()).toBe(1);
           |                                               ^
      2566 |     await expect(serverConnP).resolves.toBeUndefined();
      2567 |     // The server receives the end confirmation for graceful exit
      2568 |     await expect(serverConnEndP).resolves.toBeUndefined();

      at Object.<anonymous> (tests/network/Proxy.test.ts:2565:47)

Test Suites: 1 failed, 1 total
Tests:       3 failed, 26 passed, 29 total
Snapshots:   0 total
Time:        75.722 s, estimated 87 s
Ran all test suites matching /tests\\network\\proxy/i.
GLOBAL TEARDOWN
Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-WkqxaW
Jest did not exit one second after the test run has completed.

This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.

PS C:\Users\Emma Casolin\Projects\js-polykey> npm run test tests/nodes/nodeconnection.test -- --maxWorkers=50%

> @matrixai/polykey@1.0.0 test
> jest "tests/nodes/nodeconnection.test" "--maxWorkers=50%"

Determining test suites to run...
GLOBAL SETUP
Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-kK3jf3
WARN:ConnectionReverse 127.0.0.1:64741:Server Error: ErrorConnectionEndTimeout
WARN:ConnectionReverse 127.0.0.1:64741:Reverse Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:64742:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:64744:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionReverse 127.0.0.1:64745:Server Error: ErrorConnectionEndTimeout
ERROR:NodeConnection test:Failed CONNECT to 128.0.0.1:12345 - ErrorConnectionStart: Send failed (status: -4062)
ERROR:grpc:Failed to connect to 128.0.0.1:12345/?nodeId=vmudpqa10ulc0538eud5ae4ia913ct15sv0bhth6v85hdlgtuhea0 through proxy 127.0.0.1:52189
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52189 with error connect ECONNREFUSED 127.0.0.1:52189
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52189 with error connect ECONNREFUSED 127.0.0.1:52189
WARN:ConnectionForward 127.0.0.1:59508:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionReverse 127.0.0.1:59173:Server Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:59174:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionReverse 127.0.0.1:59175:Server Error: ErrorConnectionEndTimeout
ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:59178 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate
ERROR:grpc:Failed to connect to 127.0.0.1:59178/?nodeId=vd1g01vfel7usqkn7bijvvd8cfte7q4h0e2jfnj8rgmn66ih8f8ng through proxy 127.0.0.1:52207
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52207 with error connect ECONNREFUSED 127.0.0.1:52207
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52207 with error connect ECONNREFUSED 127.0.0.1:52207
WARN:NodeConnection test:Failed connection from 127.0.0.1:59180 - ErrorConnectionNotRunning
ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:59181 - ErrorConnectionStart: Client network socket disconnected before secure TLS co
ERROR:grpc:Failed to connect to 127.0.0.1:59181/?nodeId=vdd6vphgd4n9lmff3h0t2p5r5s27r594r1qidp26a1e56rm798b7g through proxy 127.0.0.1:52216
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52216 with error connect ECONNREFUSED 127.0.0.1:52216
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52216 with error connect ECONNREFUSED 127.0.0.1:52216
WARN:ConnectionForward 127.0.0.1:58219:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51189:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51191:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51193:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51195:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51200:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51202:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51204:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:51206:Client Error: ErrorConnectionEndTimeout
ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate
ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=viala3ulemjdt7pfchefu9khp4pub301c1dli3i5unitjp6k4h2c0 through proxy 127.0.0.1:52313
WARN:ConnectionReverse 127.0.0.1:51212:Server Error: ErrorConnectionEndTimeout
WARN:NodeConnection test:Failed connection from 127.0.0.1:51212 - ErrorConnectionNotRunning
ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorConnectionStart: Client network socket disconnected before secure TLS co
ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=viala3ulemjdt7pfchefu9khp4pub301c1dli3i5unitjp6k4h2c0 through proxy 127.0.0.1:52313
ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate
ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=viala3ulemjdt7pfchefu9khp4pub301c1dli3i5unitjp6k4h2c0 through proxy 127.0.0.1:52313
WARN:ConnectionReverse 127.0.0.1:51212:Server Error: ErrorConnectionEndTimeout
WARN:NodeConnection test:Failed connection from 127.0.0.1:51212 - ErrorConnectionNotRunning
ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorConnectionStart: Client network socket disconnected before secure TLS co
ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=v6ksrp0othl670be87gcngk7gqcvcnnd8ckvkbma9o26un1lbq0o0 through proxy 127.0.0.1:52313
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52313 with error connect ECONNREFUSED 127.0.0.1:52313
WARN:ConnectionForward 127.0.0.1:51211:Client Error: ErrorConnectionEndTimeout
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52313 with error connect ECONNREFUSED 127.0.0.1:52313
ERROR:grpc:Failed to connect to proxy 127.0.0.1:52313 with error connect ECONNREFUSED 127.0.0.1:52313
WARN:ConnectionForward 127.0.0.1:51213:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:61089:Client Error: ErrorConnectionEndTimeout
 FAIL  tests/nodes/NodeConnection.test.ts (380.382 s)
  NodeConnection test
    √ session readiness (4927 ms)
    √ connects to its target (via direct connection) (4649 ms)
    √ connects to its target but proxies connect first (3998 ms)
    √ grpcCall after connection drops (4101 ms)
    √ fails to connect to target (times out) (3278 ms)
    √ getRootCertChain (4929 ms)
    √ getExpectedPublicKey (3748 ms)
    √ should call `killSelf if connection is closed based on bad certificate (3216 ms)
    √ should call `killSelf if connection is closed before TLS is established (4122 ms)
    √ should call `killSelf if the Agent is stopped. (5684 ms)
    × should call `killSelf and throw if the server exit's during testUnaryFail (43061 ms)
    × should call `killSelf and throw if the server kill's during testUnaryFail (43529 ms)
    × should call `killSelf and throw if the server sigkill's during testUnaryFail (42579 ms)
    × should call `killSelf and throw if the server exit's during testStreamFail (43072 ms)
    × should call `killSelf and throw if the server kill's during testStreamFail (42783 ms)
    × should call `killSelf and throw if the server sigkill's during testStreamFail (43681 ms)
    √ existing connection handles a resetRootKeyPair on sending side (5289 ms)
    √ existing connection handles a renewRootKeyPair on sending side (6081 ms)
    √ existing connection handles a resetRootCert on sending side (4667 ms)
    √ existing connection handles a resetRootKeyPair on receiving side (5398 ms)
    √ existing connection handles a renewRootKeyPair on receiving side (5735 ms)
    √ existing connection handles a resetRootCert on receiving side (4061 ms)
    √ new connection handles a resetRootKeyPair on sending side (5549 ms)
    √ new connection handles a renewRootKeyPair on sending side (5000 ms)
    √ new connection handles a resetRootCert on sending side (4152 ms)
    √ new connection handles a resetRootKeyPair on receiving side (9190 ms)
    √ new connection handles a renewRootKeyPair on receiving side (5578 ms)
    √ new connection handles a resetRootCert on receiving side (6292 ms)

  ● NodeConnection test › should call `killSelf and throw if the server exit's during testUnaryFail

    spawn ts-node ENOENT



  ● NodeConnection test › should call `killSelf and throw if the server exit's during testUnaryFail

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      717 |   });
      718 |   const options = ['exit', 'kill', 'sigkill'];
    > 719 |   test.each(options)(
          |                     ^
      720 |     "should call `killSelf and throw if the server %s's during testUnaryFail",
      721 |     async (option) => {
      722 |       let nodeConnection:

      at node_modules/jest-each/build/bind.js:45:11
          at Array.forEach (<anonymous>)
      at tests/nodes/NodeConnection.test.ts:719:21
      at Object.<anonymous> (tests/nodes/NodeConnection.test.ts:66:1)

  ● NodeConnection test › should call `killSelf and throw if the server kill's during testUnaryFail

    spawn ts-node ENOENT



  ● NodeConnection test › should call `killSelf and throw if the server kill's during testUnaryFail

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      717 |   });
      718 |   const options = ['exit', 'kill', 'sigkill'];
    > 719 |   test.each(options)(
          |                     ^
      720 |     "should call `killSelf and throw if the server %s's during testUnaryFail",
      721 |     async (option) => {
      722 |       let nodeConnection:

      at node_modules/jest-each/build/bind.js:45:11
          at Array.forEach (<anonymous>)
      at tests/nodes/NodeConnection.test.ts:719:21
      at Object.<anonymous> (tests/nodes/NodeConnection.test.ts:66:1)

  ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testUnaryFail

    spawn ts-node ENOENT



  ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testUnaryFail

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      717 |   });
      718 |   const options = ['exit', 'kill', 'sigkill'];
    > 719 |   test.each(options)(
          |                     ^
      720 |     "should call `killSelf and throw if the server %s's during testUnaryFail",
      721 |     async (option) => {
      722 |       let nodeConnection:

      at node_modules/jest-each/build/bind.js:45:11
          at Array.forEach (<anonymous>)
      at tests/nodes/NodeConnection.test.ts:719:21
      at Object.<anonymous> (tests/nodes/NodeConnection.test.ts:66:1)

  ● NodeConnection test › should call `killSelf and throw if the server exit's during testStreamFail

    spawn ts-node ENOENT



  ● NodeConnection test › should call `killSelf and throw if the server exit's during testStreamFail

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      787 |     global.defaultTimeout * 2,
      788 |   );
    > 789 |   test.each(options)(
          |                     ^
      790 |     "should call `killSelf and throw if the server %s's during testStreamFail",
      791 |     async (option) => {
      792 |       let nodeConnection:

      at node_modules/jest-each/build/bind.js:45:11
          at Array.forEach (<anonymous>)
      at tests/nodes/NodeConnection.test.ts:789:21
      at Object.<anonymous> (tests/nodes/NodeConnection.test.ts:66:1)

  ● NodeConnection test › should call `killSelf and throw if the server kill's during testStreamFail

    spawn ts-node ENOENT



  ● NodeConnection test › should call `killSelf and throw if the server kill's during testStreamFail

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      787 |     global.defaultTimeout * 2,
      788 |   );
    > 789 |   test.each(options)(
          |                     ^
      790 |     "should call `killSelf and throw if the server %s's during testStreamFail",
      791 |     async (option) => {
      792 |       let nodeConnection:

      at node_modules/jest-each/build/bind.js:45:11
          at Array.forEach (<anonymous>)
      at tests/nodes/NodeConnection.test.ts:789:21
      at Object.<anonymous> (tests/nodes/NodeConnection.test.ts:66:1)

  ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testStreamFail

    spawn ts-node ENOENT



  ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testStreamFail

    thrown: "Exceeded timeout of 40000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      787 |     global.defaultTimeout * 2,
      788 |   );
    > 789 |   test.each(options)(
          |                     ^
      790 |     "should call `killSelf and throw if the server %s's during testStreamFail",
      791 |     async (option) => {
      792 |       let nodeConnection:

      at node_modules/jest-each/build/bind.js:45:11
.ts:66:1)

Test Suites: 1 failed, 1 total
Tests:       6 failed, 22 passed, 28 total
Snapshots:   0 total
Time:        380.547 s, estimated 407 s
Ran all test suites matching /tests\\nodes\\nodeconnection.test/i.
GLOBAL TEARDOWN
Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-kK3jf3

PS C:\Users\Emma Casolin\Projects\js-polykey> npm run test tests/vaults -- --maxWorkers=50%    

> @matrixai/polykey@1.0.0 test
> jest "tests/vaults" "--maxWorkers=50%"

Determining test suites to run...
GLOBAL SETUP
Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-A3ophi
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout
WARN:ConnectionForward 127.0.0.1:57148:Client Error: ErrorConnectionEndTimeout
 FAIL  tests/vaults/VaultInternal.test.ts (190.854 s)
  VaultInternal
    √ VaultInternal readiness (3186 ms)
    √ is type correct (2259 ms)
    √ creating state on disk (2050 ms)
    √ accessing a change (4033 ms)
    √ maintains data across VaultInternal instances (5703 ms)
    √ can change to the current commit (3871 ms)
    √ can change commits and preserve the log with no intermediate vault mutation (9033 ms)
    × does not allow changing to an unrecognised commit (11534 ms)
    √ can change to the latest commit (10696 ms)
    √ adjusts HEAD after vault mutation, discarding forward and preserving backwards history (10862 ms)
    √ write operation allowed (2761 ms)
    √ read operation allowed (3382 ms)
    √ concurrent write operations prevented (8170 ms)
    √ commit added if mutation in writeF (3089 ms)
    √ no commit added if no mutation in write (2326 ms)
    √ commit message contains all actions made in the commit (7572 ms)
    √ no mutation to vault when part of a commit operation fails (4227 ms)
    √ concurrent read operations allowed (3402 ms)
    √ no commit after read (3459 ms)
    √ only exposes limited commands of VaultInternal (2862 ms)
    √ cannot commit when the remote field is set (1882 ms)
    × cannot checkout old commits after branching commit (11442 ms)
    × can recover from dirty state (7216 ms)
    × clean errant commits recovering from dirty state (8184 ms)
    √ commit added if mutation in writeG (3719 ms)
    √ no commit added if no mutation in writeG (2172 ms)
    √ no mutation to vault when part of a commit operation fails in writeG (2560 ms)
    √ no commit after readG (3666 ms)
    × garbage collection (10793 ms)
    √ writeF respects read and write locking (2381 ms)
    √ writeG respects read and write locking (2610 ms)
    √ readF respects write locking (1573 ms)
    √ readG respects write locking (1656 ms)
    √ readF allows concurrent reads (1556 ms)
    √ readG allows concurrent reads (1673 ms)
    √ can create with CreateVaultInternal (2919 ms)
    × can create an existing vault with CreateVaultInternal (3390 ms)
    √ stop is idempotent (1545 ms)
    √ destroy is idempotent (1492 ms)

  ● VaultInternal › does not allow changing to an unrecognised commit

    expect(received).rejects.toThrow()

    Received promise resolved instead of rejected
    Resolved to value: undefined

      220 |       await efs.close(fd);
      221 |     });
    > 222 |     await expect(vault.version(fourthCommit)).rejects.toThrow(
          |           ^
      223 |       vaultsErrors.ErrorVaultReferenceMissing,
      224 |     );
      225 |   });

      at expect (node_modules/expect/build/index.js:128:15)
      at Object.<anonymous> (tests/vaults/VaultInternal.test.ts:222:11)

  ● VaultInternal › cannot checkout old commits after branching commit

    expect(received).rejects.toThrow()

    Received promise resolved instead of rejected
    Resolved to value: undefined

      529 |         await efs.writeFile('test4', 'testdata4');
      530 |       });
    > 531 |       await expect(() => {
          |             ^
      532 |         return vault.version(thirdCommit);
      533 |       }).rejects.toThrow();
      534 |       await expect(() => {

      at expect (node_modules/expect/build/index.js:128:15)
      at Object.<anonymous> (tests/vaults/VaultInternal.test.ts:531:13)

  ● VaultInternal › can recover from dirty state

    ErrorEncryptedFSError: ENOENT: no such file or directory, zBmuurZGqWiEge3Fnt47NGg\.git\objects

      1080 |     // Walking all objects
      1081 |     const objectPath = path.join(this.vaultGitDir, 'objects');
    > 1082 |     const buckets = (await this.efs.readdir(objectPath)).filter((item) => {
           |                      ^
      1083 |       return item !== 'info' && item !== 'pack';
      1084 |     });
      1085 |     for (const bucket of buckets) {

      at node_modules/encryptedfs/src/EncryptedFS.ts:2321:15
      at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12)
      at constructor_.garbageCollectGitObjects (src/vaults/VaultInternal.ts:1082:22)
      at constructor_.setupGit (src/vaults/VaultInternal.ts:744:9)
      at constructor_.start_ (src/vaults/VaultInternal.ts:303:5)
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.start (src/vaults/VaultInternal.ts:269:14)
      at node_modules/@matrixai/async-init/src/CreateDestroyStartStop.ts:111:24
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at Object.<anonymous> (tests/vaults/VaultInternal.test.ts:561:5)

  ● VaultInternal › clean errant commits recovering from dirty state

    ErrorEncryptedFSError: ENOENT: no such file or directory, zNiUfngckMkoinYUm43jku7\.git\objects

      1080 |     // Walking all objects
      1081 |     const objectPath = path.join(this.vaultGitDir, 'objects');
    > 1082 |     const buckets = (await this.efs.readdir(objectPath)).filter((item) => {
           |                      ^
      1083 |       return item !== 'info' && item !== 'pack';
      1084 |     });
      1085 |     for (const bucket of buckets) {

      at node_modules/encryptedfs/src/EncryptedFS.ts:2321:15
      at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12)
      at constructor_.garbageCollectGitObjects (src/vaults/VaultInternal.ts:1082:22)
      at constructor_.setupGit (src/vaults/VaultInternal.ts:744:9)
      at constructor_.start_ (src/vaults/VaultInternal.ts:303:5)
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.start (src/vaults/VaultInternal.ts:269:14)
      at node_modules/@matrixai/async-init/src/CreateDestroyStartStop.ts:111:24
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at Object.<anonymous> (tests/vaults/VaultInternal.test.ts:621:5)

  ● VaultInternal › garbage collection

    ErrorEncryptedFSError: ENOENT: no such file or directory, z5jkvSDcS3sJsjn9CaWeGcK\.git\objects

      1080 |     // Walking all objects
      1081 |     const objectPath = path.join(this.vaultGitDir, 'objects');
    > 1082 |     const buckets = (await this.efs.readdir(objectPath)).filter((item) => {
           |                      ^
      1083 |       return item !== 'info' && item !== 'pack';
      1084 |     });
      1085 |     for (const bucket of buckets) {

      at node_modules/encryptedfs/src/EncryptedFS.ts:2321:15
      at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12)
      at constructor_.garbageCollectGitObjects (src/vaults/VaultInternal.ts:1082:22)
      at Object.<anonymous> (tests/vaults/VaultInternal.test.ts:724:7)

  ● VaultInternal › can create an existing vault with CreateVaultInternal

    expect(received).toHaveLength(expected)

    Expected length: 2
    Received length: 6
    Received array:  ["zPB64HxhKtEGWXBi7pjFbuz", "zPB64HxhKtEGWXBi7pjFbuz\\.git", "zPB64HxhKtEGWXBi7pjFbuz\\data", "zXjcSjDBt7ifWjZ9RdrKp1r", "zXjcSjDBt7ifWjZ9RdrKp1r\\.git", "zXjcSjDBt7ifWjZ9RdrKp1r\\data"]

      928 |         vaultsUtils.encodeVaultId(vaultId1),
      929 |       );
    > 930 |       expect(await efs.readdir('.')).toHaveLength(2);
          |                                      ^
      931 |     } finally {
      932 |       await vault1?.stop();
      933 |       await vault1?.destroy();

      at Object.<anonymous> (tests/vaults/VaultInternal.test.ts:930:38)

 FAIL  tests/vaults/VaultManager.test.ts (197.042 s)
  VaultManager
    √ VaultManager readiness (287 ms)
    √ is type correct (79 ms)
    √ can create many vaults and open a vault (23274 ms)
    √ can rename a vault (2733 ms)
    √ can delete a vault (1432 ms)
    √ can list vaults (3214 ms)
    √ able to read and load existing metadata (14462 ms)
    √ cannot concurrently create vaults with the same name (1337 ms)
    √ can concurrently rename the same vault (1571 ms)
    √ can concurrently open and rename the same vault (1350 ms)
    √ can save the commit state of a vault (2878 ms)
    √ Do actions on a vault using `withVault` (5366 ms)
    √ handleScanVaults should list all vaults with permissions (4413 ms)
    √ ScanVaults should get all vaults with permissions from remote node (8795 ms)
    √ stopping respects locks (2402 ms)
    √ destroying respects locks (2436 ms)
    √ withVault respects locks (2532 ms)
    √ Creation adds a vault (1365 ms)
    √ Concurrently creating vault with same name only creates 1 vault (1397 ms)
    √ vaults persist (1748 ms)
    √ vaults can be removed from map (1628 ms)
    √ stopping vaultManager empties map and stops all vaults (3098 ms)
    √ destroying vaultManager destroys all data (1948 ms)
    √ withVaults should throw if vaultId doesn't exist (58 ms)
    √ generateVaultId handles vault conflicts (2085 ms)
    With remote agents
      × clone vaults from a remote keynode using a vault name (6495 ms)
      × clone vaults from a remote keynode using a vault name with no history (4797 ms)
      √ fails to clone non existing vault (4147 ms)
      × clone and pull vaults using a vault id (6223 ms)
      √ should reject cloning when permissions are not set (4435 ms)
      × should reject Pulling when permissions are not set (5079 ms)
      × can pull a cloned vault (6172 ms)
      × manage pulling from different remotes (7080 ms)
      √ able to recover metadata after complex operations (16020 ms)
      × throw when trying to commit to a cloned vault (6939 ms)
      √ test pulling a vault that isn't remote (4135 ms)
      × pullVault respects locking (5953 ms)

  ● VaultManager › With remote agents › clone vaults from a remote keynode using a vault name

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)
      at src/vaults/VaultManager.ts:627:23

  ● VaultManager › With remote agents › clone vaults from a remote keynode using a vault name with no history

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)
      at src/vaults/VaultManager.ts:627:23

  ● VaultManager › With remote agents › clone and pull vaults using a vault id

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
          at runMicrotasks (<anonymous>)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)

  ● VaultManager › With remote agents › should reject Pulling when permissions are not set

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
          at runMicrotasks (<anonymous>)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)

  ● VaultManager › With remote agents › can pull a cloned vault

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
          at runMicrotasks (<anonymous>)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)

  ● VaultManager › With remote agents › manage pulling from different remotes

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
          at runMicrotasks (<anonymous>)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)

  ● VaultManager › With remote agents › throw when trying to commit to a cloned vault

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
          at runMicrotasks (<anonymous>)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)

  ● VaultManager › With remote agents › pullVault respects locking

    ErrorPolykeyRemote: Ref HEAD cannot be found

      201 |       if (key === 'UNKNOWN' && errorData != null) {
      202 |         const error: Error = JSON.parse(errorData, reviver);
    > 203 |         const remoteError = new grpcErrors.ErrorPolykeyRemote(
          |                             ^
      204 |           metadata,
      205 |           error.message,
      206 |           {

      at toError (src/grpc/utils/utils.ts:203:29)
      at gf (src/grpc/utils/utils.ts:453:13)
          at runMicrotasks (<anonymous>)
      at constructor_.request (src/vaults/VaultInternal.ts:783:22)
      at src/vaults/VaultInternal.ts:164:55
      at src/nodes/NodeConnectionManager.ts:204:16
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12)
      at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36)

 FAIL  tests/vaults/utils.test.ts
  Vaults utils
    × EFS can be read recursively (393 ms)
    √ fs can be read recursively (7 ms)
    √ decodeNodeId does not throw an error (3 ms)

  ● Vaults utils › EFS can be read recursively

    EBUSY: resource busy or locked, unlink 'C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-8gaaEG\LOCK'



 FAIL  tests/vaults/VaultOps.test.ts (172.971 s)
  VaultOps
    √ adding a secret (3108 ms)
    √ adding a secret and getting it (3070 ms)
    × able to make directories (3745 ms)
    √ adding and committing a secret 10 times (37850 ms)
    √ updating secret content (3456 ms)
    √ updating secret content within a directory (4506 ms)
    √ updating a secret 10 times (11808 ms)
    × deleting a secret (7323 ms)
    × deleting a secret within a directory (1778 ms)
    √ deleting a secret 10 times (18010 ms)
    √ renaming a secret (3594 ms)
    × renaming a secret within a directory (4516 ms)
    × listing secrets (7718 ms)
    √ listing secret directories (4938 ms)
    × adding hidden files and directories (4922 ms)
    × updating and deleting hidden files and directories (12287 ms)
    × adding a directory of 1 secret (2530 ms)
    √ adding a directory with subdirectories and files (4681 ms)
    √ testing the errors handling of adding secret directories (6501 ms)
    × adding a directory of 100 secrets with some secrets already existing (23817 ms)

  ● VaultOps › able to make directories

    expect(received).toContain(expected) // indexOf

    Expected value: "dir-3"
    Received array: ["dir-1", "dir-2", "dir-3\\dir-4", "dir-3\\dir-4\\secret-1"]

      124 |       expect(dir).toContain('dir-1');
      125 |       expect(dir).toContain('dir-2');
    > 126 |       expect(dir).toContain('dir-3');
          |                   ^
      127 |
      128 |       expect(await efs.readdir('dir-3')).toContain('dir-4');
      129 |       expect(await efs.readdir(path.join('dir-3', 'dir-4'))).toContain(

      at tests/vaults/VaultOps.test.ts:126:19
      at src/vaults/VaultInternal.ts:427:14
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:122:5)

  ● VaultOps › deleting a secret

    expect(received).rejects.toThrow()

    Received promise resolved instead of rejected
    Resolved to value: undefined

      204 |     );
      205 |     await vaultOps.deleteSecret(vault, 'secret-1');
    > 206 |     await expect(() => vaultOps.deleteSecret(vault, 'dir-1')).rejects.toThrow();
          |           ^
      207 |     await vaultOps.deleteSecret(vault, path.join('dir-1', 'secret-2'));
      208 |     await vaultOps.deleteSecret(vault, 'dir-1');
      209 |     await expect(vault.readF((efs) => efs.readdir('.'))).resolves.not.toContain(

      at expect (node_modules/expect/build/index.js:128:15)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:206:11)

  ● VaultOps › deleting a secret within a directory

    expect(received).rejects.toThrow()

    Received promise resolved instead of rejected
    Resolved to value: undefined

      212 |   });
      213 |   test('deleting a secret within a directory', async () => {
    > 214 |     await expect(() =>
          |           ^
      215 |       vaultOps.mkdir(vault, path.join('dir-1', 'dir-2')),
      216 |     ).rejects.toThrow(errors.ErrorVaultsRecursive);
      217 |     await vaultOps.mkdir(vault, path.join('dir-1', 'dir-2'), {

      at expect (node_modules/expect/build/index.js:128:15)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:214:11)

  ● VaultOps › renaming a secret within a directory

    expect(received).resolves.toContain(expected) // indexOf

    Expected value: "secret-change"
    Received array: []

      268 |       path.join(dirPath, 'secret-change'),
      269 |     );
    > 270 |     await expect(vault.readF((efs) => efs.readdir(dirPath))).resolves.toContain(
          |                                                                       ^
      271 |       `secret-change`,
      272 |     );
      273 |   });

      at Object.toContain (node_modules/expect/build/index.js:194:22)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:270:71)

  ● VaultOps › listing secrets

    expect(received).toStrictEqual(expected) // deep equality

    - Expected  - 1
    + Received  + 1

      Array [
    -   "dir1/dir2/secret-3",
    +   "dir1\\dir2\\secret-3",
        "secret-1",
        "secret-2",
      ]

      281 |       'secret-content',
      282 |     );
    > 283 |     expect((await vaultOps.listSecrets(vault)).sort()).toStrictEqual(
          |                                                        ^
      284 |       ['secret-1', 'secret-2', 'dir1/dir2/secret-3'].sort(),
      285 |     );
      286 |   });

      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:283:56)

  ● VaultOps › adding hidden files and directories

    ErrorEncryptedFSError: ENOENT: no such file or directory, .hiddenDir\.hiddenInSecret

      67 |   for (const dirent of dirents) {
      68 |     const res = path.join(dir, dirent.toString());
    > 69 |     const stat = await fs.promises.stat(res);
         |                  ^
      70 |     if (stat.isDirectory()) {
      71 |       yield* readdirRecursively(fs, res);
      72 |     } else if (stat.isFile()) {

      at node_modules/encryptedfs/src/EncryptedFS.ts:2932:15
      at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12)
      at readdirRecursively (src/vaults/utils.ts:69:18)
      at Object.readdirRecursively (src/vaults/utils.ts:71:7)
      at src/vaults/VaultOps.ts:256:22
      at src/vaults/VaultInternal.ts:427:14
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at Object.listSecrets (src/vaults/VaultOps.ts:254:10)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:328:18)

  ● VaultOps › updating and deleting hidden files and directories

    ErrorEncryptedFSError: ENOENT: no such file or directory, .hidingDir\.hiddenInSecret

      67 |   for (const dirent of dirents) {
      68 |     const res = path.join(dir, dirent.toString());
    > 69 |     const stat = await fs.promises.stat(res);
         |                  ^
      70 |     if (stat.isDirectory()) {
      71 |       yield* readdirRecursively(fs, res);
      72 |     } else if (stat.isFile()) {

      at node_modules/encryptedfs/src/EncryptedFS.ts:2932:15
      at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12)
      at readdirRecursively (src/vaults/utils.ts:69:18)
      at Object.readdirRecursively (src/vaults/utils.ts:71:7)
      at src/vaults/VaultOps.ts:256:22
      at src/vaults/VaultInternal.ts:427:14
      at withF (node_modules/@matrixai/resources/src/utils.ts:24:12)
      at Object.listSecrets (src/vaults/VaultOps.ts:254:10)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:351:18)

  ● VaultOps › adding a directory of 1 secret

    expect(received).resolves.toContain(expected) // indexOf

    Expected value: "secret"
    Received array: []

      380 |     await expect(
      381 |       vault.readF((efs) => efs.readdir(secretDirName)),
    > 382 |     ).resolves.toContain('secret');
          |                ^
      383 |
      384 |     await fs.promises.rm(secretDir, {
      385 |       force: true,

      at Object.toContain (node_modules/expect/build/index.js:194:22)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:382:16)

  ● VaultOps › adding a directory of 100 secrets with some secrets already existing

    expect(received).resolves.toContain(expected) // indexOf

    Expected value: "secret 0"
    Received array: []

      512 |         await expect(
      513 |           vault.readF((efs) => efs.readdir(secretDirName)),
    > 514 |         ).resolves.toContain('secret ' + j.toString());
          |                    ^
      515 |       }
      516 |       expect(
      517 |         (

      at Object.toContain (node_modules/expect/build/index.js:194:22)
      at Object.<anonymous> (tests/vaults/VaultOps.test.ts:514:20)

Test Suites: 4 failed, 4 total
Tests:       24 failed, 75 passed, 99 total
Snapshots:   0 total
Time:        364.67 s
Ran all test suites matching /tests\\vaults/i.
GLOBAL TEARDOWN
Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-A3ophi

[emmacasolin]

Fixing the failures mentioned in this issue needs to be done before working on the integration tests for these platforms, since there are likely underlying issues we aren't aware of and that would make it difficult to work on integration tests. The failures can be grouped into a few different categories:

1. ENOENT - We seem to frequently see ENOENT errors when attempting to spawn a child process on Windows. We also sometimes see ENOENT being returned from operations using the EFS (also on Windows) and when attempting to write to files (e.g. keys) on Mac - potentially related to this ci: merge staging to master js-db#38 (comment)
2. Pathnames - This one is a very simple issue, and it may even be the underlying cause of a lot of other failures on Windows. This refers to Windows using \ to separate pathnames rather than /
3. Ref HEAD cannot be found - This could potentially be related to the issue with path names on Windows using \ instead of /, however we often see this error when performing vaults operations on remote agents
4. Permissions - An example of this is Error: EPERM: operation not permitted, unlink 'C:\Users\GITLAB~1\AppData\Local\Temp\polykey-test-m4rbmK\token' in the sessions tests (Windows)
5. EBUSY - There seem to be some locking issues on Windows, for example when writing to keypairs, e.g. EBUSY: resource busy or locked, unlink 'C:\Users\GITLAB~1\AppData\Local\Temp\polykey-test-4f3f1G\db\000005.ldb'
6. Timeouts - Almost all of the failures on Mac are timeouts, however, some of these timeouts persist even when only a single test is run, so I think something else is going on here that causes these timeouts.

[CMCDragonkai]

I've found additional problems on windows with respect to process.stdout.write and console programs.

There are 3 dimensions to dealing with Windows for console programs.

1. Asynchronous vs synchronous - https://nodejs.org/api/process.html#a-note-on-process-io
2. Encoding - we want to standardise on UTF-8
3. Line Endings - posix uses LF, windows uses CRLF, we can use os.EOL to auto switch

I've found that powershell and cmd also behave differently.

I've written this test script:

const os = require('os');
const process = require('process');
const fs = require('fs');
const { Buffer } = require ('buffer');

async function main(argv = process.argv) {
  argv = argv.slice(2);
  if (argv[0] === '0') {
    console.log('Hello 盘');
  } else if (argv[0] === '1') {
    process.stdout.write(`HELLO${os.EOL}盘${os.EOL}`);
  } else if (argv[0] === '2') {
    // LF
    process.stdout.write('HELLO\n盘\n');
  } else if (argv[0] === '3') {
    // CRLF
    process.stdout.write('HELLO\r\n盘\r\n');
  } else if (argv[0] === '4') {
    // WITH setting the utf-8 encoding
    process.stdout.setDefaultEncoding('utf-8');
    // process.stdout.setEncoding('utf-8');
    process.stdout.write('HELLO\n盘\n');
  } else if (argv[0] === '5') {
    // WITH setting the utf-8 encoding
    process.stdout.setDefaultEncoding('utf-8');
    process.stdout.write('HELLO\r\n盘\r\n');
  } else if (argv[0] === '6') {
    // Writing buffers directly
    process.stdout.write(Buffer.from('hello\n盘\n', 'utf-8'));
  } else if (argv[0] === '7') {
    // Writing buffers directly
    process.stdout.write(Buffer.from('hello\r\n盘\r\n', 'utf-8'));
  } else if (argv[0] === '8') {
    // Setting encoding AND writing buffers
    process.stdout.setDefaultEncoding('utf-8');
    // process.stdout.setEncoding('utf-8');
    process.stdout.write(Buffer.from('hello\n盘\n', 'utf-8'));
  } else if (argv[0] === '9') {
    // Setting encoding AND writing buffers
    process.stdout.setDefaultEncoding('utf-8');
    process.stdout.write(Buffer.from('hello\r\n盘\r\n', 'utf-8'));
  } else if (argv[0] === '10') {
    process.stdout.write(
      Buffer.from('hello\n盘\n', 'utf-8'),
      'utf-8'
    );
  } else if (argv[0] === '11') {
    // Writing with fs
    fs.writeFileSync(1, 'hello\n盘\n');
  } else if (argv[0] === '12') {
    // Writing with fs
    fs.writeFileSync(1, 'hello\r\n盘\r\n');
  } else if (argv[0] === '13') {
    // Writing with fs and specifying encoding
    fs.writeFileSync(1, 'hello\n盘\n', 'utf-8');
  } else if (argv[0] === '14') {
    // Writing with fs and specifying encoding
    fs.writeFileSync(1, 'hello\r\n盘\r\n', 'utf-8');
  } else if (argv[0] === '15') {
    // Writing with fs and buffers
    fs.writeFileSync(1, Buffer.from('hello\n盘\n', 'utf-8'));
  } else if (argv[0] === '16') {
    // Writing with fs and buffers
    fs.writeFileSync(1, Buffer.from('hello\r\n盘\r\n', 'utf-8'));
  } else if (argv[0] === '17') {
    // Writing to a file
    fs.writeFileSync('./lf.txt', 'hello\n盘\n');
    fs.writeFileSync('./crlf.txt', 'hello\r\n盘\r\n');
  }
}

void main();

Then I've ran node ./test.js X > ./tmp/X.txt and vary the X from 0 to 16, then run node ./test.js 17.

Or use this script on powershell.

for (($i = 0); $i -lt 17; $i++) { node ./test.js $i > ./tmp/$i.txt }
node ./test.js 17

The results show this:

Powershell ALWAYS converts everything to UTF-16 LE with CRLF, no exceptions, the only case where this doesn't occur is with 17, where nodejs directly writes to files.

»» ~/Desktop
 ♖ file *.txt                                                                                                       (master) pts/2 15:52:01
0.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
10.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
11.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
12.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
13.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
14.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
15.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
16.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
1.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
2.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
3.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
5.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
6.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
7.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
8.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
9.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
10.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
crlf.txt: Unicode text, UTF-8 text, with CRLF line terminators
lf.txt:   Unicode text, UTF-8 text

CMD doesn't actually support unicode characters on render, and it can't even show them anyway. So the Chinese character becomes just a question mark box. But what about LF/CRLF and encoding?

Well it turns out, that CMD preserves unicode encoding during piping. It does not convert it to UTF-16 LE. It also preserves line endings, it does not auto convert LF to CRLF. Basically it does what Linux does, and it is far more obvious about the behaviour.

»» ~/Desktop/cmd
 ♖ file *.txt                                                                                                       (master) pts/2 15:55:53
0.txt:    Unicode text, UTF-8 text
10.txt:   Unicode text, UTF-8 text
11.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
12.txt:   Unicode text, UTF-8 text
13.txt:   Unicode text, UTF-8 text
14.txt:   Unicode text, UTF-8 text, with CRLF line terminators
15.txt:   Unicode text, UTF-8 text
16.txt:   Unicode text, UTF-8 text, with CRLF line terminators
1.txt:    Unicode text, UTF-8 text, with CRLF line terminators
2.txt:    Unicode text, UTF-8 text
3.txt:    Unicode text, UTF-8 text, with CRLF line terminators
4.txt:    Unicode text, UTF-8 text
5.txt:    Unicode text, UTF-8 text, with CRLF line terminators
6.txt:    Unicode text, UTF-8 text
7.txt:    Unicode text, UTF-8 text, with CRLF line terminators
8.txt:    Unicode text, UTF-8 text
9.txt:    Unicode text, UTF-8 text, with CRLF line terminators
crlf.txt: Unicode text, UTF-8 text, with CRLF line terminators
lf.txt:   Unicode text, UTF-8 text

[CMCDragonkai]

Let's figure out a principle of least surprise here.

At first, I was thinking that we should use os.EOL to ensure that when printing to a terminal, we are always outputting the OS-specific line endings.

However if someone is working across platforms, then it can be quite surprising if users expect that a text file that they output/pipe contains CRLF while normal file writes from PK creates LF line endings. This is because we do tell users to use > and direction operators to write to files with our CLI.

We would preserve whatever line endings the user uses when they write the file to PK.

At the same time, it appears windows CMD and powershell understand \n and will create newlines.

Therefore:

1. We should continue to use \n and not os.EOL when outputting anything on the terminal. Both CMD and powershell will render it as a newline.
2. We should preserve the input newlines whatever they are if they are CRLF or LF. This means when outputting or writing to a file, we will preserve whatever binary data that was inputted in the first place. This should have integration tests checking this. If users get confused, we would ask them to ensure that they are using LF line endings to maintain compatibility... and we can eventually add an option to auto-convert CRLF to LF or LF to CRLF...

However we still have a problem:

1. Is there a way to get CMD to render characters?
2. Is there a way to tell powershell NOT to auto convert to CRLF, and not to auto convert to UTF-16?

Now according to https://nodejs.org/api/fs.html#fswritefd-string-position-encoding-callback

On Windows, if the file descriptor is connected to the console (e.g. fd == 1 or stdout) a string containing non-ASCII characters will not be rendered properly by default, regardless of the encoding used. It is possible to configure the console to render UTF-8 properly by changing the active codepage with the chcp 65001 command. See the chcp docs for more details.

But of course this would be an instruction to end users to set this up. Let me check.

[CMCDragonkai]

Using chcp 65001 is actually not enough. One must also be using a font that supports unicode characters. I switched to lucida console from the default "Consolas", but again it is not capable of displaying Chinese characters.

These 2 Q&A show how complex it is for windows to setup to properly use UTF-8:

• https://stackoverflow.com/questions/49476326/displaying-unicode-in-powershell
• https://stackoverflow.com/questions/57131654/using-utf-8-encoding-chcp-65001-in-command-prompt-windows-powershell-window/57134096#57134096

[CMCDragonkai]

Using intl.cpl with:

That's sufficient to change to chcp 65001 by default across the entire opreating system. This could be recommended to end users... but again could be quite problematic if the rest of the operating system isn't ready.

At the same time, one must choose a correct font. Luicda Console or the default Consolas is fine for displaying some unicode, but only SimSun-ExtB can do Chinese characters.

The font selection doesn't appear to affect SSHing into the powershell. That probably relies on whatever chcp it is. So at the end of the day, users may recommended to use chcp 65001 or stick it into their profile.

THIS however does not fix the problem of piping. Still piping into any file proceeds to convert LF to CRLF and converts to UTF 16 LE.