julep: "chain of custody" error handling

[StefanKarpinski]

A of the major issues with try/catch is that it's possible to catch the wrong error and think that you are handling a mundane, expected problem, when in fact something far worse is wrong and the problem is an unexpected one that should terminate your program. Partly for this reason, Julia discourages the use of try/catch and tries to provide APIs where you can check for situations that may cause problems without actually raising an exception. That way try/catch is really left primarily as a way to trap all errors and make sure your program continues as best it can.

This is also why we have not introduced any sort of typed catch clause into the language. This proposal introduces typed catch clauses, but they only catch errors of that type under certain circumstances in a way that's designed to avoid the problem of accidentally catching errors from an unexpected location that really should terminate the program. Here's what a Java-inspired typed catch clause might look like in Julia:

try
    # nonsense
    foo(1,2,3) # throws BazError
    # shenanigans
catch err::BazError
    # deal with BazError only
end

Of course, we don't have this in Julia, but it can be simulated like this:

try
    # nonsense
    foo(1,2,3) # throws BazError
    # shenanigans
catch err
    isa(err, BazError) || rethrow(err)
    # deal with BazError only
end

Although this does limit the kinds of errors one might catch, it fundamentally doesn't solve the problem, which is that while there are some places where you may be expecting a BazError to be thrown, other code that you're calling inside the try block might also throw a BazError where you didn't expect it, in which case you shouldn't try to handle because you'll just be covering up a real, unexpected problem. This problem is especially bad in the generic programming contexts where you don't really know what kind of code a given function call might end up running.

To address this, I'm proposing adding a hypothetical throws keyword that allows you to annotate function calls with a type that they're expected to throw and which you can catch with a type catch clause (it's a little more subtle than this, but bear with me). The above example would be written like this:

try
    # nonsense
    foo(1,2,3) throws BazError
    # shenanigans
catch err::BazError
    # deal with *expected* BazError only
end

The only difference is that the throws BazError comment after the call to foo became syntax. So the question is what does this annotation do? The answer is that without that annotation indicating that you expect the expression to throw an error of type BazError, you can't catch it with a typed catch. In the original version where the throws BazError was just a comment, the typed catch block would not catch a BazError thrown by the call to foo – because the lack of annotation implies that such an error is unxepected.

There is a bit more, however: the foo function also has to annotate the point where the BazError might come from. So, let's say you had these two definitions:

Consider something like this:

function foo1(x,y,z)
    bar(x,2y) throws BazError
end

function foo2(x,y,z)
    bar(x,2y)
end

function bar(a,b)
    throw BazError()
end

This also introduce a hypothetical keyword form of throw – more on that below. These definitions result in the following behavior:

try
    # nonsense
    foo1(1,2,3) throws BazError
    # shenanigans
catch err::BazError
    # error from bar is caught
end

try
    # nonsense
    foo2(1,2,3) throws BazError
    # shenanigans
catch err::BazError
    # error from bar is NOT caught
end

The rule is that a typed catch only catches the an error if every single call site from the current function down to the one that actually throws the error is annotated with throws ErrorType where the actual thrown error object is an instance of ErrorType. An untyped catch still catches all errors, leaving the existing behavior unchanged:

try foo1(1,2,3)
catch
    # error is caught
end

try foo2(1,2,3)
catch
    # error is caught
end

So what is the rationale behind this proposal and why is it any better than just having typed catch clauses? The key point is that in order to do a typed catch, there has to be a "chain of custody" from the throw site all the way up to the catch site, and each step has to expect getting the kind of type that's thrown. There are two ways to catch the wrong error:

1. catch an error that was not the type of error you were expecting
2. catch an error that came from a place you weren't expecting it to come from

The first kind of mistake is prevented by giving an error type, while the second kind of mistake is prevented by the "chain of custody" between the throw site and the catch site.

Another way of thinking about this is that the throws ErrorType annotations are a way of making what exceptions a function throws part of its official type behavior. This is akin to how Java puts throws ErrorType after the signature of the function. But in Java it's a usability disaster because you are required to have the throws annotation as soon as you do something that could throw some kind of non-runtime exception. In practice, this is so annoying that it's pretty common to just raise RuntimeErrors instead of changing all the type signatures in your whole system. Instead of making runtime excpetion vs. non-runtime exception a property of the error types as Java does, this proposal makes it a property of how the error occurs. If an error is occurs in an expected way, then it's a non-runtime exception and you can catch it with a typed catch clause. If an error occurs in an unexpected way, then it's a runtime exception and you can only catch it with an untyped catch clause. You can only catch errors by type if they are part of the "official behavior" of the function you're calling, where official behavior is indicated with throws ErrorType annotations.

One detail of this proposal that I haven't mentioned yet is that throw would become a keyword instead of just a function. The reason for this is that writing

throw(BazError()) throws BazError

seems awfully verbose and redundant. For compatibility, we could allow throw() to still invoke the function throw while throw ErrorType(args...) would be translated to

throw(ErrorType(args...)) throws ErrorType

This brings up another issue – what scope should the right-hand-side of throws be evaluated in? In one sense, it really only makes sense to evaluate it in the same scope that the function signature is evalutated in. However, this is likely to be confusing since all other expressions inside of function bodies are evaluated in the local scope of the function. It would be feasible to do this too, and just rely on the fact that most of the time it will be possible to statically evaluate what type expression produces. Of course, when that isn't possible, still be necessary to emit code that will do the right thing depending on the runtime value of the expression. If the r-h-s is evaluated in local scope, then we can say that throws expr is equivalent to this:

throw(expr) throws typeof(expr)

Usually it will be possible to staticly determine what typeof(expr) is, and it is a simpler approach to explain than restricting the syntax of the expression after the throw keyword.

Note that under this proposal, these are not the same:

try
    # stuff
catch
    # catches any error
end

try
    # stuff
catch ::Exception
    # only catches "official" errors
end

Also note that this proposal is, other than syntax additions that are not likely to cause big problems, completely backwards compatible: existing untyped catch clauses continue to work the way they used to – they catch everything. It would only be new typed catch clauses that only catch exceptions that have an unbroken "chain of custody".

[tknopp]

+1 for the syntax addition.

I have to admit that I am not sold on the idea of expected and unexpected exceptions. I would find it rather "unexpected" if I indicate to catch something that is then not catched ;-)

For me the real issue in such situations i that the exception hierarchy is not well designed and/or that the wrong exception is used from the exception hierarchy.

[StefanKarpinski]

The problem with typed exceptions is that they give you a sense of precision, but they're not really precise at all. Making the exception part of the type of a function is a good idea – that part Java got right. What's not so good is forcing the caller to handle every kind of exception a function it uses can throw.

Here's a motivating scenario: let's say you implement a new kind of AbstractVector – say a kind of sparse vector or something – let's call it SpArray. Internally, it stores values in normal Arrays. Someone is using this and finds that sometimes they end up making indexing errors – they use a typed catch block to handle this. But you've made a programming error in implementing the SpArray type and it's actually SpArray that's encountering the indexing error internally because of a mistake. But this is caught and treated as if was a user-level indexing error. With this proposal, that situation won't occur because at the point where the SpArray implementation is incorrect and causes an indexing error, there is no throws BoundsError annotations so the resulting exception can't be caught with a typed catch clause.

[tknopp]

I absolutely see your point that it can lead to hard to find bugs if one try/catches too generic exceptions and I have been running into this myself in C++ a lot. ("Oh this throws exceptions all the time. Put try { ... } catch(...) {}around it and problem solved").

To play around with your example maybe this should be handled by throwing SpArrayBoundsError in SpArray and ArrayBoundsError in Array. Of course both deriving from BoundsError. In this way one can catch the SpArray specific bounds errors and let all other BoundsError pass through.

[StefanKarpinski]

To play around with your example maybe this should be handled by throwing SpArrayBoundsError in SpArray and ArrayBoundsError in Array. Of course both deriving from BoundsError. In this way one can catch the SpArray specific bounds errors and let all other BoundsError pass through.

That's a common solution offered in languages with exception hierarchies and typed catch, but I don't buy it. The logical extreme of that approach is to have an error type for almost every single location where an exception can be thrown. At that point, what you're really doing is labeling individual exception locations and using labels to catch things – albeit labels that are carefully arranged into a hierarchy. That need for a carefully arranged hierarchy to make the approach tenable is also fishy – and easily abused. I came up with this approach by thinking about how to compromise between a broad classification scheme with a set of standard exception types and individual labels for exception locations.

[tknopp]

Yes you are absolutely right. Exception hierarchies have the potential to get very specific up to the point that specific lines get specific error types. But as one subtypes from broader exception types I don't see the issue with that. But maybe I just have bad taste to not think this is fishy :-)
So please lets look what others think about this.

[JeffBezanson]

Talking about this as "labeling locations" gets me thinking: would it be possible just to label the locations? Every exception could consist of an exception object and a label (symbol). The SpArray library could throw BoundsErrors, labeling them as :SpBoundsError. You could catch by type and/or label.

[aviks]

You're right that using typed exceptions correctly in Java is annoying. Everyone throws RuntimeException subclases, and I've trained myself to see the Exception type as primarily informational, useful only for logging.

But my fear is, annotating exceptional (?) call sites all the way down is going to turn out to be equally tedious. Is this something that is easy enough to reason about and implement, so that it will be widely practiced?

[StefanKarpinski]

@JeffBezanson – I thought about just labeling locations, but it seems too granular. Sometimes different locations throw the same error, no? Also, how is that different from having an exception type for every location? I guess you wouldn't have to declare the types, but you also wouldn't get any type hierarchy. How would labels be scoped? By module?

[JeffBezanson]

Yes, you'd probably want some kind of scoping, which remains to be worked out. But different locations could in fact throw the same error; in my overly-simple formulation they could just use the same symbol.

The idea is that exception and origin are orthogonal: what and where-from. "What" is the exception type hierarchy, which describes problems, and "where-from" is the locations, which might follow the module hierarchy, or have no hierarchy.

[StefanKarpinski]

@aviks – I think this should actually be pretty rare. Having a "throws" annotation is essentially saying "throwing this error is part of the official API of this function". Currently we basically consider all exceptions to be runtime errors, so anything is real problem and not part of the functions official behavior. That would continue to be the default. Only when you decide that something like BoundsErrors or KeyError is part of the official interface of AbstractArray or Associative would you put throws BoundsErrors and throws KeyError annotations on function calls. And hopefully you wouldn't need very many of them. I should probably try out adding these annotations for something like that and see how it goes.

[tknopp]

The labeling idea is interesting. Still it increases the dimensionality of the dispatch mechanism from a one-dimensional to a two-dimensional thing.

I am all for putting more context into exceptions. In C# for instance one usually also gets the line numbers where the exception was thrown. Further when debugging exceptions I found it very useful to have exception chains, i.e. when an exception is rethrown (as a different exception type) the original exception is put as an "inner exception" to the outer exception.

My own experience is that in small projects and research code exceptions are mainly an error mechanism for debugging "not yet correct" code. In these situations one will hardly use try/catch and reason about what exception to catch. The intensive use of the error function in Julia base is an indicator that this is what most people use exceptions for. The importance for exceptions increased a lot for me when working on large projects and projects where others are the users (i.e. production code). In these cases one has to prevent by all means that exceptions remain uncatched and thus has to maintain a sensible exception hierarchy and use try/catch at various locations. But this is also usually nothing where the system exception hierarchy is used. Instead I want to throw TheUSBCableIsUnplugged exceptions that are carefully catches and translated to ThePrinterIsNotAvailable and so on. And for this purpose the Julia exception system is already working quite well.

[samoconnor]

I am in the process of porting my Tcl AWS library to Julia as a way to learn Julia, (and so that I can then experiment with implementing projects reliant on AWS in Julia).

In a first-pass of the Julia docs, I saw try/catch/finally and made a mental check "that is supported".
When it comes to writing actual code I am surprised that try/catch doesn't do what I expect. Most Julia features have either made me think "nice, that's how it should be done", or "that's weird" followed by some reading followed by "that's nice". The design of Julia's try/catch seems to be that it is deliberately somewhat crippled to try to wean people off using it. This smells wrong to me.

In distributed, networked systems individual nodes and connections can fail there are propagation delays, data models are eventually-consistent. Clear, robust, exception handling is a must. I want to write code that assumes everything is reliable, immediately globally updated etc and deal with the exceptional glitches in one place in high-level retry or conflict resolution loops.

I am not a fan of type-based exception handling, having had the misfortune of working with Java etc. Tcl has no types, so exceptions are trapped based by matching an error-code prefix. An error-code in Tcl is just a list.

e.g.

proc get {url} {
    ...
    throw {HTTP 404}  "Error: $url Not Found"
    ...
    throw [list HTTP 300 $location] "Error: $url has moved to $location"
    ...
}

try {

   get $url

} trap {HTTP 404} {} {
    puts "get can't find $url"
} trap {HTTP 300} {message info} {
    set location [lindex $info 2]
    get $location
}

# Ignore missing dict key...
proc lazy_get {dict key} {
    try {

        dict get $mydict "key"

    } trap {TCL LOOKUP DICT} {} {}
}

# Catch exit status of shell command...
retry count 2 {

     exec ./create_queue.tcl "foobar"

 } trap EX_TEMPFAIL {} {
    after 60000
}

Every error (exception) in a Tcl program is supposed to have a human-readable error message and a machine-readable errorcode.

The source of an error can include as much or as little machine-readable info as it likes.

Safe exception handling is achieved by trapping only sufficiently specific errorcodes. Of course you can shoot yourself in the foot by writing "trap {}" if you want to...

http://www.tcl.tk/man/tcl8.6/TclCmd/try.htm
http://www.tcl.tk/man/tcl8.6/TclCmd/throw.htm

I think using types to match exceptions isn't great. But since Julia matches methods by type as a core feature, perhaps it makes sense for Julia. I would prefer something like passing a Dict() to throw and having a convenient syntax for catching only exceptions that match a dictionary pattern.

The key thing is readability of the try/catch code.

Below are some fragments of Tcl code that I am face with porting to Julia.
I am open to the idea that there is just a better way to do these things in Julia. But on the other hand, I think these are reasonable uses of an exception mechanism...

try {

    set web_user [create_aws_iam_user $aws $region-web-user]

} trap EntityAlreadyExists {} {
    delete_aws_iam_user_credentials $aws $region-web-user
    set web_user [create_aws_iam_user_credentials $aws $region-web-user]
}

    retry count 4 {

        set res [aws_sqs $aws CreateQueue QueueName $name {*}$attributes]

    } trap QueueAlreadyExists {} {

        delete_aws_sqs_queue [aws_sqs_queue $aws $name]

    } trap AWS.SimpleQueueService.QueueDeletedRecently {} {

        puts "Waiting 1 minute to re-create SQS Queue \"$name\"..."
        after 60000
    }

proc fetch {bucket key {byte_count {}}} {

    if {$byte_count ne {}} {
        set byte_count [list Range bytes=0-$byte_count]
    }
    try {

        s3 get $::aws $bucket $key {*}$byte_count

    } trap NoSuchKey {} {
    } trap AccessDenied {} {}
}

proc aws_ec2_associate_address {ec2 elastic_ip} {

    puts "Assigning Elastic IP: $elastic_ip"
    retry count 3 {

        aws_ec2 $ec2 AssociateAddress InstanceId [get $ec2 id] \
                                      PublicIp $elastic_ip

    } trap InvalidInstanceID {} {
        after [expr {$count * $count * 1000}]
    }
}

proc create_aws_s3_bucket {aws bucket} {

    try {

        aws_rest $aws PUT $bucket Content-Type text/plain Content [subst {
                <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
                  <LocationConstraint>[aws_bucket_region $bucket]</LocationConstraint>
                </CreateBucketConfiguration>
        }]

    } trap BucketAlreadyOwnedByYou {} {}
}

proc delete_aws_sqs_queue {queue} {

    try {

        aws_sqs $queue DeleteQueue

    } trap AWS.SimpleQueueService.NonExistentQueue {} {}
}

proc aws_s3_exists {aws bucket path} {

    try {

        aws_s3_get $aws $bucket $path Range bytes=0-0
        return 1

    } trap NoSuchKey {} {
    } trap AccessDenied {} {}
    return 0
}

    retry count 3 {

        return [$command $aws {*}$args]

    } trap {TCL LOOKUP DICT AWSAccessKeyId} {} {

        set aws [get_aws_ec2_instance_credentials $aws]

    } trap ExpiredToken {message info} {

        if {![exists ::oc_aws_ec2_instance_credentials]} {
            return -options $info $message
        }

        puts "Refreshing EC2 Instance Credentials..."
        set aws [get_aws_ec2_instance_credentials $aws -force-refresh]
    }

    try {

        aws_iam $aws DeleteInstanceProfile InstanceProfileName $name

    } trap NoSuchEntity {} {}

    set response [aws_iam $aws CreateInstanceProfile \
                               InstanceProfileName $name \
                               Path $path]

[hayd]

other code that you're calling inside the try block might also throw a BazError where you didn't expect it

This seems like an edge case, if you keep the try block small (e.g. just the line you're expecting a failure) it shouldn't be the case... if it's critically different why not subclass BazError?

+1, a syntax for typed exceptions would be great. IMO isa(err, BazError) ... is quite messy. I like:

catch err::BazError
    # deal with *expected* BazError only
catch err::FooError
    # deal with *expected* FooError only
end

[elextr]

if you keep the try block small (e.g. just the line you're expecting a failure) it shouldn't be the case...

The syntax proposed by @StefanKarpinski is really just a more compact syntactic sugar for a try block around the important line (or is it intended to apply to an expression?) which passes the exception to the outer catch.

if it's critically different why not subclass BazError?

You can't when its thrown by a library you called, not your code.

The concept of identifying several small regions where "it is understood that this may throw xxxerror, and I'm prepared to handle it in a common handler" seems a good idea, but as currently proposed this is likely to surprise those coming from other languages (who don't RTFM because they expect all languages to be the same :)

This also doesn't provide for one of the traditional use-cases where you want to catch some types of errors no matter where they occur, but let the others go:

 try
    a big fast but flakey algorithm
 catch err::matherror # oh no underflowed, overflowed or divided by zero
    alternative slow algorithm
 end