Julia Client

[Beforerr]

I implemented a working Julia client a while ago and was wondering if it is possible/worthwhile to migrate the repo into the Org.

Repo: https://github.com/Beforerr/HAPIClient.jl

Related: JuliaSpacePhysics/HAPIClient.jl#1

[rweigel]

We are happy to see this, and we'll look into it and get back to you.

One thing we do to test clients is loop through the "TestData" servers and verify that they can read the response: https://github.com/hapi-server/client-matlab/blob/master/hapi_test.m

[Beforerr]

I do have a limited test here https://github.com/Beforerr/HAPIClient.jl/blob/main/test/data.jl .

Would add more tests later.

[rweigel]

We'll also probably want to create a script template so the code can be cut and pasted, e.g., https://hapi-server.org/servers/#server=CDAWeb&dataset=A2_K0_MPA&parameters=dens_lop&start=2003-10-29T00:03:04Z&stop=2003-10-31T00:03:04.000Z&return=script

We do some unconventional things in the scripts, such as calling the OS to install dependencies (for example, in the Python script) or downloading dependencies (for example, in the MATLAB script). This is so things will "just work," and we figure that an experienced user will remove the unconventional parts and do things the right way after they are convinced things work as expected without having to expend more effort than a cut-and-paste.

[Beforerr]

This is pretty cool. Just wondering how the template is created (like in what language) so I can help with Julia's one?

[rweigel]

The templates are in https://github.com/hapi-server/server-ui/tree/master/scripts

Client-side Jascript code does replacements of capitalized terms in the web UI (But there is server-side code that uses the same Javascript code: https://hapi-server.org/servers/api/)

I'd also recommend creating a DOI for the repository.

[Beforerr]

Hi @rweigel , Thanks for the pointers! I’ve now registered the package in the Julia registry, and there’s a DOI badge in the README for easy referencing. I also just added a template in test/scripts.jl if you’d like to take a look. Let me know if you have any feedback on the template or on the overall approach!

[rweigel]

Thanks again for sharing this with us!

---

This looks fairly close to being general enough so that it can be made available as a generated script, but I may want to do a bit more testing. I'll look at this more over the next week or so.

---

In looking at integrating the template, I noticed that

start_ = "2001-01-01T05:00:00Z"
end_ = "2001-01-01T06:00:00Z"
data = hapi(CDAWeb, dataset, parameters, start_, end_)

Does not work unless the trailing Z is removed.

---

What are your thoughts on adding

if !("HAPIClient" in keys(Pkg.dependencies()))
	Pkg.add("HAPIClient")
end

---

I am new to Julia, so one of the first things I tried was to show the data values. I was also curious about how to show the numbers in the console and plot the data. Copilot didn't suggest anything that worked. I'll probably research this more and suggest additions to the README example.

---

When I saw

catalog = hapi(CDAWeb)

in the demo, I expected the code not to run because it did not seem like CDAWeb had been defined. It seems to be set at

CDAWeb = Server(; url="https://cdaweb.gsfc.nasa.gov/hapi", id="CDAWeb", title="CDAWeb")
CSA = Server(; url="https://csatools.esac.esa.int/HapiServer/hapi", id="CSA", title="Cluster Science Archive", HAPI="3.2", format="csv")

This is somewhat risky because these URLs may change as may the HAPI version. We recommend using https://raw.githubusercontent.com/hapi-server/servers/refs/heads/master/servers.json to find the URL associated with an ID.

[Beforerr]

Hi @rweigel Thanks for the detailed feedback! About:

1. Datetime Parsing:
   I’ve updated the datetime parsing so it now correctly handles ISO8601 strings with the trailing “Z”. This should resolve the issue you mentioned with the start and end times.

2. Dependency Check:
   I think that checking if "HAPIClient" is already in Pkg.dependencies() isn’t necessary since Pkg.add is fast and won’t cause any issues if the package is already installed.

3. Server Defaults:
   The servers now default to using JSON from the URL you suggested. This change should help avoid hard-coding URLs and HAPI versions, making the code more robust against changes.

4. Data:

   Data could be showed using var.value. This is also Tab completion when typing var.

   

5. Visualization:

   Right now it is possible with another package using Makie (codes is fairly simple but the package has a lot of other feathers thus heavy deps) , see https://beforerr.github.io/SPEDAS.jl/dev/tutorials/getting-started/

Thanks again for sharing your insights. Let me know if you have any further suggestions or run into any other issues!

[rweigel]

Using commit 2c4660fe7ecb50a1ed5aec3503e95762993fd049

using HAPIClient

# Retrieve data for specific parameters within a time range
parameters = "Magnitude,BGSEc"
start_time = "2001-01-01T05:00:00Z"
end_time = "2001-01-01T06:00:00Z"
data = hapi(CDAWeb, dataset, parameters, start_time, end_time)

println(data[1].values)

Gives

ERROR: LoadError: ArgumentError: Invalid DateTime string
Stacktrace:
 [1] parse(::Type{DateTime}, s::String, df::DateFormat{Symbol("yyyy-mm-dd\\THH:MM:SS.s"), Tuple{…}})
   @ Dates /Applications/Julia-1.11.app/Contents/Resources/julia/share/julia/stdlib/v1.11/Dates/src/parse.jl:278
 [2] DateTime (repeats 2 times)
   @ /Applications/Julia-1.11.app/Contents/Resources/julia/share/julia/stdlib/v1.11/Dates/src/io.jl:568 [inlined]
 [3] get_data(server::HAPIClient.Server, dataset::String, parameters::String, tmin::String, tmax::String; format::String)
   @ HAPIClient ~/.julia/packages/HAPIClient/7dbN2/src/data.jl:13
 [4] get_data
   @ ~/.julia/packages/HAPIClient/7dbN2/src/data.jl:10 [inlined]
 [5] hapi(server::HAPIClient.Server, dataset::String, parameters::String, tmin::String, tmax::String)
   @ HAPIClient ~/.julia/packages/HAPIClient/7dbN2/src/HAPIClient.jl:27
 [6] top-level scope
   @ ~/git/hapi/HAPIClient.jl/demo.jl:13
in expression starting at /Users/rweigel/git/hapi/HAPIClient.jl/demo.jl:13
Some type information was truncated. Use `show(err)` to see complete types.

[Beforerr]

This is weird. Using latest commit on my machine, it worked.

[rweigel]

Hmm. This works, so it may be due to the trailing Z.

Note that the HAPI spec requires a trailing Z, but the clients have been written to accept strings without it. Another complication is the HAPI spec technically requires trailing Z even for YYYY, YYYY-MM, YYYY-MM-DD, and YYYY-DOY even though this is not part of ISO8601 (and the HAPI spec claims it uses a subset of ISO8601; a trailing Z on dates only it is a part of the XML datetime standard and we mistakenly made it a part of the requirement).

Also, if I set parameters="", I get an error. The script generation code will use parameters="" if the user requests all parameters. I can make a pull request to address this, if desired.

using HAPIClient

# Define server, dataset, parameters, and time range
server = "http://hapi-server.org/servers/TestData2.0/hapi"
dataset = "dataset1"
parameters = "scalar"
start_time = "1970-01-01"
stop_time = "1970-01-01T00:00:11"

data = hapi(server, dataset, parameters, start_time, stop_time)

[Beforerr]

The related codes is in https://github.com/Beforerr/HAPIClient.jl/blob/main/src/specs/time.jl, i should now cover all the usages you mentioned.

Oh, I do not know 'parameters=""' is a feature. Yes go ahead, you are 100% welcome to make a pull request

[rweigel]

Here is the test we use in Python: https://github.com/hapi-server/client-python/blob/master/hapiclient/test/test_hapitime2datetime.py#L59

[Beforerr]

Just added more test, and parsing is right

https://github.com/Beforerr/HAPIClient.jl/blob/main/test/runtests.jl

[Beforerr]

Hi @rweigel , I just want to follow up if there is any plan to put the Julia Client on WebUI. Some of my colleagues are using it so it might be helpful to have a Web script template to just copy and paste.

[rweigel]

I'll get back to you in about a week.

[rweigel]

I did not see that you had added tests for all of the test data servers (this is what I was waiting for).

Here is a suggested demo.jl to put in the root dir (so users don't need to copy/paste from README). I removed the CSA example b/c that server is often slow.

Is it possible to provide more feedback about the steps that are being taken with print statements? Also, is there a way to do a very basic plot (without a new dependency) of the output at the end?

Note that I removed your alternative call option (SERVER/DATASET/PARAMETERS) because DATASET can have forward slashes, as can PARAMETERS.

Thanks again for sharing this!

I'll start working on a script. However, I'll need an example where you use a URL for a server rather than an id such as "CDAWeb".

using Pkg
Pkg.add("HAPIClient")

using HAPIClient
using Dates

# List available HAPI servers
servers = hapi()

# Use a predefined server (CDAWeb or CSA)
# Get catalog of available datasets from CDAWeb
catalog = hapi(CDAWeb)

# Get information about parameters in a dataset
dataset = "AC_H0_MFI"
params = hapi(CDAWeb, dataset)

# Retrieve data for specific parameters within a time range
parameters = "Magnitude,BGSEc"
start_time = DateTime(2001, 1, 1, 5, 0, 0)
end_time = DateTime(2001, 1, 1, 6, 0, 0)
data = hapi(CDAWeb, dataset, parameters, start_time, end_time)

# Access the data
Magnitude = data[1]
BGSEc = data[2]

[Beforerr]

Hi Bob, thanks for the suggestions and feedback!

Just to clarify, the added tests in test/data.jl cover all the test data servers—similar in structure to the original MATLAB test. Everything under the test folder is tested, including those using real URLs, so feel free to take a look and let me know if anything seems missing.

Regarding the basic plot: since Julia recommend to avoid unnecessary dependencies by design, we would need to bring in a plotting package (like Makie.jl or Plots.jl) and enable it via package extension. I can definitely add that if desired, though it does mean an extra install step for users.

Could you elaborate what do you mean "Is it possible to provide more feedback about the steps that are being taken with print statements?". Do you mean something like logging the download process?

[rweigel]

Just to clarify, the added tests in test/data.jl cover all the test data servers—similar in structure to the original MATLAB test. Everything under the test folder is tested, including those using real URLs, so feel free to take a look and let me know if anything seems missing.

Indeed, I saw those updates. Somehow, I missed them earlier. I was waiting for them to be complete before proceeding.

For the demo, perhaps put a conditional that checks to see if Makie or Plots is installed and if so, executes the plot? And in a comment state how to do the install. Perhaps this would be obvious to regular Julia users, but in my case it would have helped.

Regarding feedback, when I ran the demo from the command line, it executed and finished silently. It was not clear to me that something worked. So perhaps "Getting data from {URL}" and then "Got {n} parameters" would do it. This is not that important, but will help if something goes wrong - I could try the URL in a browser and see if there was an error. In the python client there is an option "logging" that provides feedback on what is happening in the background if set to true.

[Beforerr]

Now there is a verbose option for feedback (it is passed to HTTP.get, default is 0 (other options: 1, 2).) JuliaSpacePhysics/HAPIClient.jl#11

[rweigel]

I've added a client-julia repo to https://github.com/hapi-server/.

Next steps are

1. Add reference to hapi-server.org
2. Add auto script generation to hapi-server.org/servers
3. Look for list of programming languages in docs and add Julia

[rweigel]

I've updated https://hapi-server.org/ and https://hapi-server.org/servers to mention Julia

and one can now have a Julia script created using the HAPI Data Explorer: example

Many thanks to @Beforerr for this.

@Beforerr Have you ever considered presenting at the https://dash2025.space.swri.edu/ meeting? It seem many of your projects at https://github.com/JuliaSpacePhysics would be of interest to the DASH community.

[Beforerr]

Thank you too for your efforts @rweigel ! I wasn’t aware of the meeting before — it does look like a very interesting one. (Unfortunately, it seems I’ve already missed the abstract submission deadline:)

[jvandegriff]

The deadline just recently passed, and you could very likely still get something in if you submit in the next day or two. It is still a pretty informal meeting. (I'm on the organizing committee... :)