Not using "strong zeros" for deriatives of constant values

[MasonProtter]

Compare:

julia> using Diffractor, Zygote

julia> var"'"(f) = Zygote.var"'"(f);

julia> exp'(Inf)
Inf

julia> exp''(Inf)
Inf

with

julia> var"'"(f) = Diffractor.PrimeDerivativeFwd(f);

julia> exp'(Inf)
Inf

julia> exp''(Inf)
NaN

You can see in the code_typed that it appears to be subtracting 0.0 * exp(x)^2 from the answer of 1.0 * exp(x), causing the NaN. Ideally this'd instead be 1.0 * exp(x) - ZeroTangent()*exp(x)^2 so that we don't get the NaN.

[oxinabox]

We used to have this.
We broke this in #189 in order to get type stability.
Because we wanted branches that returned the same type on each side to also have the same partial on each side.
And this is not the case if branches with literals on one side return strong ZeroTangent but branches with nonliteral values do not. e.g relu(x) = x > 0 ? 0.0 : x

Not having that type stability is in theory fine, but in practice the small unions optimization often failed, causing 100x slowdown.
I would like to have our cake and eat it too, but I am not sure how.
One possibility is special inference pass, which @Keno and I talked about at JuliaCon.