Instrumenting jobs

[areina]

Hello @msolli, thanks for your work on this library.

Did you think about adding monitoring/instrumentation to the jobs? Although I understand it might be possible to wrap the handle-job! multimethod with some logic to produce metrics, it would be great to get that behavior by default. What's your take on this?

[msolli]

Hi @areina!

I might be inclined to something with OpenTelemetry, but so far I've covered my use cases by wrapping the enqueue! and handler-fn! functions.

Do you have any thoughts on what tracing and metrics behaviors should be included by default? I see that you're with NewRelic, which means you probably have more experience in this area than me.

Here's what we're doing at work to instrument the workers:

;; Enqueue

(defn with-trace-data
  [payload]
  (let [trace-context (clj-otel.context/->headers)]
    (cond-> payload
      (seq trace-context) (assoc ::trace-context trace-context))))

(defn enqueue!
  ([conn queue job-type payload]
   (assert (map? payload) "Worker job payload must be a map")
   (trace.span/with-span! {:name       (str `enqueue!)
                           :span-kind  :producer
                           :attributes {:queue    queue
                                        :job-type job-type
                                        :payload  payload}}
     (let [job-id (job/enqueue! conn job-type (with-trace-data payload) {:proletarian/queue      queue
                                                                         :proletarian/serializer serializer})]
       (trace.span/add-span-data! {:attributes {:job-id job-id}})
       (log/info (str `enqueue! " " job-type) {:queue queue :job-type job-type :payload payload :job-id job-id})
       job-id))))


;; Handle job

(defmulti handle-job! (fn [job-type _payload] job-type))

(defn handle-job-wrapper!
  [job-type payload]
  (let [trace-context (some-> (::trace-context payload) (clj-otel.context/headers->merged-context))
        payload       (dissoc payload ::trace-context)]
    (trace.span/with-span! {:name       (str job-type)
                            :parent     trace-context
                            :attributes {:payload payload}
                            :span-kind  :consumer}
      (log/info (str `handle-job! " " job-type) {:job-type job-type, :payload payload})
      (handle-job! job-type payload))))

Here I'm using the https://github.com/steffan-westcott/clj-otel library. This produces spans like this:

(Yes, we're using NewRelic! :) )

[areina]

Going with OpenTelemetry makes total sense to me. At this moment in a side project I was just reporting these four metrics:

• :proletarian/job_count (counter)
• :proletarian/job_success (counter)
• :proletarian/job_failed (counter)
• :proletarian/job_duration (histogram)

I took a look at what other similar projects were doing and I found this for sidekiq (ruby): https://github.com/fastly/sidekiq-prometheus

I thought it could be cool to have this behavior by default, or something that you can enable when you configure the worker. However, as a first step could be great just to add your examples to the documentation.

Thanks!

[fuadsaud]

I would be really interested in getting a nice setup for this. For example, I'd like to publish a single event (log/metric) after the handler finishes with the following attributes:

• hander execution duration
• status (success, error)
• attempt count
• queue name

I guess I can wrap the handler and try to reconstruct some of these from the context (e.g. wrap with try catch to detect an error), but some of the information (like attempt count) wouldn't be available from that context.

Perhaps that could be done by hooking into the log function and generating a new type of log event based on the incoming log event?