Skip to content

Fix remote spec handling and hash calculation #3826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 6, 2019
Merged

Fix remote spec handling and hash calculation #3826

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
acacd35
1792 fix remote spec handling and hash calculation (#3440)
Sep 4, 2019
042e176
This patch fixes the bug that we cannot access to remote files when c…
fuj1g0n Sep 4, 2019
a1ecb15
fix detecting remote file / local file logic while finding the hash f…
fuj1g0n Sep 5, 2019
6ab750f
add testcase
fuj1g0n Sep 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions modules/openapi-generator-maven-plugin/examples/java-client.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
<!-- /RELEASE_VERSION -->
<executions>
<execution>
<id>default</id>
<goals>
<goal>generate</goal>
</goals>
Expand All @@ -39,6 +40,36 @@
<library>jersey2</library>
</configuration>
</execution>
<execution>
Copy link
Contributor Author

@fuj1g0n fuj1g0n Sep 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the testcase you mentioned #3826 (comment).
But I couldn't reproduce the error in my environment.

mvn clean compile -f modules/openapi-generator-maven-plugin/examples/java-client.xml

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.openapitools:sample-project >-------------------
[INFO] Building sample-project 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ sample-project ---
[INFO] Deleting /mnt/c/publicgithub/openapi-generator/modules/openapi-generator-maven-plugin/examples/target
[INFO]
[INFO] --- openapi-generator-maven-plugin:4.1.2-SNAPSHOT:generate (default) @ sample-project ---
[WARNING] Output directory does not exist, or is inaccessible. No file (.openapi-generator-ignore) will be evaluated.
[INFO] OpenAPI Generator: java (client)
[INFO] Generator 'java' is considered stable.
......
......
[INFO] --- openapi-generator-maven-plugin:4.1.2-SNAPSHOT:generate (remote) @ sample-project ---
[WARNING] Output directory does not exist, or is inaccessible. No file (.openapi-generator-ignore) will be evaluated.
[INFO] OpenAPI Generator: java (client)
[INFO] Generator 'java' is considered stable.
......
......
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ sample-project ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /mnt/c/publicgithub/openapi-generator/modules/openapi-generator-maven-plugin/examples/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ sample-project ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 46 source files to /mnt/c/publicgithub/openapi-generator/modules/openapi-generator-maven-plugin/examples/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49.616 s
[INFO] Finished at: 2019-09-06T16:24:45+09:00
[INFO] ------------------------------------------------------------------------

Copy link
Contributor Author

@fuj1g0n fuj1g0n Sep 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check your error will be reproduced with my latest commit including testcase?

mvn clean install
mvn clean compile -f modules/openapi-generator-maven-plugin/examples/java-client.xml

Copy link
Contributor Author

@fuj1g0n fuj1g0n Sep 6, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also waiting for how Shippable CI will judge the testcase.

Copy link
Member

@wing328 wing328 Sep 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pull the latest again and no longer seeing the issue. Must be the setup in my end.

Thanks for adding the test case.

I'll merge after all CI tests pass.

<id>remote</id>
<phase>generate-sources</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<!-- specify the swagger yaml -->
<inputSpec>https://raw.githubusercontent.com/OpenAPITools/openapi-generator/master/modules/openapi-generator/src/test/resources/2_0/petstore.yaml</inputSpec>

<!-- target to generate java client code -->
<generatorName>java</generatorName>

<!-- hint: if you want to generate java server code, e.g. based on Spring Boot,
you can use the following target: <generatorName>spring</generatorName> -->

<!-- pass any necessary config options -->
<configOptions>
<dateLibrary>joda</dateLibrary>
</configOptions>

<!-- override the default library to jersey2 -->
<library>jersey2</library>

<output>${project.build.directory}/generated-sources/remote-openapi</output>
<apiPackage>remote.org.openapitools.client.api</apiPackage>
<modelPackage>remote.org.openapitools.client.model</modelPackage>
<invokerPackage>remote.org.openapitools.client</invokerPackage>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
Expand Down
95 changes: 84 additions & 11 deletions ...api-generator-maven-plugin/src/main/java/org/openapitools/codegen/plugin/CodeGenMojo.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,18 @@
import static org.apache.commons.lang3.StringUtils.isNotEmpty;
import static org.openapitools.codegen.config.CodegenConfiguratorUtils.*;

import io.swagger.v3.parser.core.models.AuthorizationValue;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.net.URLConnection;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.channels.ReadableByteChannel;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
Expand All @@ -42,6 +53,7 @@
import org.openapitools.codegen.CodegenConfig;
import org.openapitools.codegen.CodegenConstants;
import org.openapitools.codegen.DefaultGenerator;
import org.openapitools.codegen.auth.AuthParser;
import org.openapitools.codegen.config.CodegenConfigurator;
import org.openapitools.codegen.config.GlobalSettings;
import org.sonatype.plexus.build.incremental.BuildContext;
Expand Down Expand Up @@ -429,7 +441,7 @@ public void execute() throws MojoExecutionException {
if (inputSpecFile.exists()) {
File storedInputSpecHashFile = getHashFile(inputSpecFile);
if(storedInputSpecHashFile.exists()) {
String inputSpecHash = Files.asByteSource(inputSpecFile).hash(Hashing.sha256()).toString();
String inputSpecHash = calculateInputSpecHash(inputSpecFile);
String storedInputSpecHash = Files.asCharSource(storedInputSpecHashFile, Charsets.UTF_8).read();
if (inputSpecHash.equals(storedInputSpecHash)) {
getLog().info(
Expand Down Expand Up @@ -720,12 +732,7 @@ public void execute() throws MojoExecutionException {

// Store a checksum of the input spec
File storedInputSpecHashFile = getHashFile(inputSpecFile);
ByteSource inputSpecByteSource =
inputSpecFile.exists()
? Files.asByteSource(inputSpecFile)
: CharSource.wrap(ClasspathHelper.loadFileFromClasspath(inputSpecFile.toString().replaceAll("\\\\","/")))
.asByteSource(Charsets.UTF_8);
String inputSpecHash =inputSpecByteSource.hash(Hashing.sha256()).toString();
String inputSpecHash = calculateInputSpecHash(inputSpecFile);

if (storedInputSpecHashFile.getParent() != null && !new File(storedInputSpecHashFile.getParent()).exists()) {
File parent = new File(storedInputSpecHashFile.getParent());
Expand All @@ -746,8 +753,75 @@ public void execute() throws MojoExecutionException {
}
}

/**
* Calculate openapi specification file hash. If specification is hosted on remote resource it is downloaded first
*
* @param inputSpecFile - Openapi specification input file to calculate it's hash.
* Does not taken into account if input spec is hosted on remote resource
* @return openapi specification file hash
* @throws IOException
*/
private String calculateInputSpecHash(File inputSpecFile) throws IOException {

URL inputSpecRemoteUrl = inputSpecRemoteUrl();

File inputSpecTempFile = inputSpecFile;

if (inputSpecRemoteUrl != null) {
inputSpecTempFile = File.createTempFile("openapi-spec", ".tmp");

URLConnection conn = inputSpecRemoteUrl.openConnection();
if (isNotEmpty(auth)) {
List<AuthorizationValue> authList = AuthParser.parse(auth);
for (AuthorizationValue auth : authList) {
conn.setRequestProperty(auth.getKeyName(), auth.getValue());
}
}
ReadableByteChannel readableByteChannel = Channels.newChannel(conn.getInputStream());

FileOutputStream fileOutputStream = new FileOutputStream(inputSpecTempFile);
FileChannel fileChannel = fileOutputStream.getChannel();

fileChannel.transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
}

ByteSource inputSpecByteSource =
inputSpecTempFile.exists()
? Files.asByteSource(inputSpecTempFile)
: CharSource.wrap(ClasspathHelper.loadFileFromClasspath(inputSpecTempFile.toString().replaceAll("\\\\","/")))
.asByteSource(Charsets.UTF_8);

return inputSpecByteSource.hash(Hashing.sha256()).toString();
}

/**
* Try to parse inputSpec setting string into URL
* @return A valid URL or null if inputSpec is not a valid URL
*/
private URL inputSpecRemoteUrl(){
try {
return new URI(inputSpec).toURL();
} catch (URISyntaxException | MalformedURLException | IllegalArgumentException e) {
Copy link
Contributor Author

@fuj1g0n fuj1g0n Sep 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is to get the URL if remote otherwise null.
when inputSpec is local file and relative path, IllegalArgumentException is thrown.
In this case, we can return null because this method is expected to return URL only when the inputSpec is remote.

return null;
}
}

/**
* Get specification hash file
* @param inputSpecFile - Openapi specification input file to calculate it's hash.
* Does not taken into account if input spec is hosted on remote resource
* @return a file with previously calculated hash
*/
private File getHashFile(File inputSpecFile) {
return new File(output.getPath() + File.separator + ".openapi-generator" + File.separator + inputSpecFile.getName() + ".sha256");
String name = inputSpecFile.getName();

URL url = inputSpecRemoteUrl();
if (url != null) {
String[] segments = url.getPath().split("/");
name = Files.getNameWithoutExtension(segments[segments.length - 1]);
}

return new File(output.getPath() + File.separator + ".openapi-generator" + File.separator + name + ".sha256");
}

private String getCompileSourceRoot() {
Expand All @@ -757,8 +831,7 @@ private String getCompileSourceRoot() {
final String sourceFolder =
sourceFolderObject == null ? "src/main/java" : sourceFolderObject.toString();

String sourceJavaFolder = output.toString() + "/" + sourceFolder;
return sourceJavaFolder;
return output.toString() + "/" + sourceFolder;
}

private void addCompileSourceRootIfConfigured() {
Expand Down Expand Up @@ -803,4 +876,4 @@ private void adjustAdditionalProperties(final CodegenConfig config) {
}
}
}
}
}