Support for TS 3 project references

[VojtaStanek]

We have a monorepo when one of the packages is compiled with webpack, but references other package from the monorepo. We are using project references (new in TS 3) to compile only the packages, that are needed (the changed one and other that have reference to it). Is there a way to make it work with webpack? Currently we are compiling it with tsc and then bundling compiled js with webpack, but it is not an ideal solution, since many features doesn't work so well (e.g. source maps and custom loaders such as sass-loader) and it is quite slow.

[johnnyreilly]

Hey @cactucs,

Project references support isn't implemented as yet. If you'd like to we'd be happy to look at a PR. If you're wondering what would need to change then take a look at the gulp-typescript changes as a guide:

https://github.com/ivogabe/gulp-typescript/pull/579/files

Don't worry about broken tests; feel free to just start the PR process and we can use that as a basis for collaboration 😄

[andrewbranch]

@johnnyreilly I started taking a look at this; I got curious about it at the exact same time as @cactucs. It seems that ts-loader’s transpileOnly path is pretty similar to what gulp-typescript does, but the code paths that use the language service and the experimental watch API are pretty different.

As a test, I wrote a project where the root tsconfig file uses strict: false and the referenced project tsconfig uses strict: true, and contains some code that has a semantic error due to strict mode. I did figure out how to get getProjectReferences onto the typescript.LangaugeServiceHost, but the compiler doesn’t come up with the error in languageService.getSemanticDiagnostics(). I stepped into the compiler source itself and verified that the program instance does indeed have program.getProjectReferences() that seems to be working correctly.

I'll fork and push what I've done so far.

[andrewbranch]

Here's the diff: https://github.com/TypeStrong/ts-loader/compare/master...andrewbranch:project-references?expand=1

[andrewbranch]

That said, I'm not sure what exactly the expected behavior for webpack and project references is. tsc --build works in part by checking timestamps on the emitted JS and d.ts files from dependent composite projects, but of course using webpack is going to totally change what happens to that emit (gets piped into the bundler instead of written to disk right away). At least, semantic error checking should work against any composite projects’ specific tsconfig, but otherwise, how do project references change what this loader is supposed to do?

[johnnyreilly]

First of all, thanks for your work @andrewbranch!

That said, I'm not sure what exactly the expected behavior for webpack and project references is

An excellent question; and honestly I'm not sure I know the answer.

I'll cc @RyanCavanaugh @rbuckton in case they have a view. But for myself I'm not too sure.

That said, I'm open to exploring options.

[andrewbranch]

Just pushed a commit that “fixes” making the language service aware of project references, causing an error to occur when the dependent composite project isn’t built (which is indeed what tsc does without --build), so that seems like a step forward. Need to update the error reporting hooks so that the error that’s shown isn’t

Error: Could not find file: '/Users/Andrew/Developer/webpack-dependent-projects-test/src/shared/index.ts'.
    at getValidSourceFile (/Users/Andrew/Developer/ts-loader/node_modules/typescript/lib/typescript.js:111577:23)
    at Object.getSyntacticDiagnostics (/Users/Andrew/Developer/ts-loader/node_modules/typescript/lib/typescript.js:111765:52)
    at provideErrorsToWebpack (/Users/Andrew/Developer/ts-loader/dist/after-compile.js:113:36)
    at /Users/Andrew/Developer/ts-loader/dist/after-compile.js:23:9

but rather the semantic diagnostic generated by the program, which you see with tsc:

error TS6305: Output file '/Users/Andrew/Developer/webpack-dependent-projects-test/dist/index.d.ts' has not been built from source file '/Users/Andrew/Developer/webpack-dependent-projects-test/src/shared/index.ts'.
src/index.ts(1,25): error TS2307: Cannot find module './shared'.

As I'm beginning to understand more how this works, it seems like ts-loader could take one (or either/both with a loader option) of two positions:

1. ts-loader just works like running tsc. Existing built composite projects will be used instead of recompiling the source files, which does have the advantage of being faster, not including the time spent outside of webpack building those composite projects. I think getting this far is pretty easy—similar to what gulp-typescript added in the PR you linked.
2. ts-loader works like running tsc --build, rebuilding composite projects when files inside those projects change. I have no idea how difficult that would be; I skimmed through some TypeScript APIs and couldn’t find if or where that orchestrator functionality is exposed. I think this would be useful, though, as it would eliminate the need for adding more complexity to our existing development pipeline (I guess we’d have to spin up a separate watch task to watch the dependent projects if we wanted to keep our current behavior of getting a rebuild when changing any referenced file, even ones that we don’t change that often, which we’d like to put in a composite project.

[johnnyreilly]

it seems like ts-loader could take one (or either/both with a loader option) of two positions

Please keep investigating. I'm open to having an option that flips between modes if that seems useful. Probably best you do some real world experiments and report back on what seems to make sense.

[andrewbranch]

Sounds good to me; I'm happy to keep experimenting, but would definitely appreciate input from the TS team here. Technically I guess they’re coworkers of mine so maybe I can track someone down. 😄

Pushed another commit that basically finishes option 1 for the language service path; should hopefully be simple enough to do the same for the createProgram and createWatchProgram paths.

I've been testing and debugging ad-hoc for this so far; the test directory here is daunting. Any tips on where to start if I were to begin writing some real tests?

Edit: also, do you have a preference on how to do TypeScript version detection? We can only call program.getProjectReferences() on >= 3.0.

[johnnyreilly]

Sounds good to me; I'm happy to keep experimenting, but would definitely appreciate input from the TS team here.

Great! @mhegazy @DanielRosenwasser @sheetalkamat @RyanCavanaugh @rbuckton would any one of you be able to advise your erstwhile colleague @andrewbranch on what a sensible course of action here would look like?

I've been testing and debugging ad-hoc for this so far; the test directory here is daunting. Any tips on where to start if I were to begin writing some real tests?

Yeah it is a bit. Essentially there's 2 test suites; each with a subtly different purpose. The comparison test pack and the execution test pack. Have a read of the README's to find out more about them:

https://github.com/TypeStrong/ts-loader/blob/master/test/comparison-tests/README.md
https://github.com/TypeStrong/ts-loader/blob/master/test/execution-tests/README.md

The TL;DR is this: the comparison test pack captures file outputs for future diffing. It's useful for checking files are generated in response to various actions (you have fake files being updated and capture the resulting output). The execution test pack actually executes tests written in TypeScript, and transpiled into JS. These tests are more reliable but don't provide any insight as to what triggers file generation.

Probably given your use case a comparison test will be most appropriate. But I know these are daunting and I'm happy to pitch in and help guide you.

Edit: also, do you have a preference on how to do TypeScript version detection? We can only call program.getProjectReferences() on >= 3.0.

Probably just doing program.getProjectReferences && program.getProjectReferences() would work? I don't feel super strongly. Aim for readability and simplicity but otherwise go nuts 😄

cc @TheLarkInn

[andrewbranch]

@johnnyreilly, how would you feel about splitting my proposed two possibilities into two PRs? I think I’m almost done with the first part (using built project references, but not building them), which I think is useful on its own, and I would hate to see it languish if figuring out the second part takes a long time or I can’t commit as much time in the future. I'll go ahead and open a WIP PR and start trying to figure out the tests.

[andrewbranch]

I’ve hit a bit of a speed bump upon realizing that program.getSourceFile() for a file in a project reference, assuming the dependent project is built, returns a SourceFile object describing the .d.ts declaration for that file/project. For Webpack’s purposes though, we need to get the corresponding .js file, and I’m having trouble figuring out an API to do that. Although, I think it’s guaranteed that the JS file will be colocated and share the same name as the declaration file... but that feels a bit sketchy to me...?

[johnnyreilly]

I think it’s guaranteed that the JS file will be colocated and share the same name as the declaration file... but that feels a bit sketchy to me...?

This is true unless they've set declarationDir in their tsconfig.json I believe; see https://github.com/TypeStrong/ts-loader/blob/master/src/tsconfig.json for an example.

So you should be able to derive the location of the original source file using the info in the tsconfig.json. it seems reasonable enough to try and use that as the basis for a decision?

[RyanCavanaugh]

Hey all, thanks for taking this up. Here to answer some questions (I'll be checking in on this all day so let's get chatty).

To compute the output JS filename, you can use this code from src/compiler/tsbuild.ts:

    function getOutputJavaScriptFileName(inputFileName: string, configFile: ParsedCommandLine) {
        const relativePath = getRelativePathFromDirectory(rootDirOfOptions(configFile.options, configFile.options.configFilePath!), inputFileName, /*ignoreCase*/ true);
        const outputPath = resolvePath(configFile.options.outDir || getDirectoryPath(configFile.options.configFilePath!), relativePath);
        const newExtension = fileExtensionIs(inputFileName, Extension.Json) ? Extension.Json :
                             fileExtensionIs(inputFileName, Extension.Tsx) && configFile.options.jsx === JsxEmit.Preserve ? Extension.Jsx : Extension.Js;
        return changeExtension(outputPath, newExtension);
    }

Some general discussion - in a way, project references and tsbuild's behavior of building upstream projects are actually totally orthogonal, though obviously related. Project references are just an advanced form of path mapping, and tsbuild's build ordering/dependency algorithm is selected mostly for simplicity.

The choice of building referenced projects or not, and the choice of what the right up-to-date check, really depends on the tool and I'm honestly not well-informed enough about Webpack to make the right call on whether there's one right answer or space for configurability here.

[andrewbranch]

@RyanCavanaugh awesome, thanks! Appreciate you taking the time to offer some expertise here.

One immediate question: looking at the snippet from tsbuild.ts you pasted, that function isn’t exported and most of the functions it calls are also not exported—would the team consider exporting this so we don’t have to duplicate the logic?