DSP Hardware Acceleration API

DSP offload client (SDK-2).

Thin wrapper over the camera-daemon DSP service (platform PLAT-1..6): buffers are allocated on the FD-publisher UDS (/run/aipc/camera.sock, FD_PUB_MSG_DSP_ALLOC/RESP/BUF_RELEASE, dma-buf fds via SCM_RIGHTS) and jobs are submitted through the SubmitDspJob gRPC on camera-control.

Hardware-first with a numpy CPU fallback: on a daemon without the DSP RPC (grpc UNIMPLEMENTED) or with the service not running (error -5) the *_hw helpers compute the result on CPU instead of raising, and client.last_used_hw records which path served the last call.

Caveat (P0 platform contract): a job source must be a daemon-registered dma-buf, so a plain numpy array is copied into one. Zero-copy input IS available for camera frames: pass a Frame received with keep_fd=True` (or its ``.handle) and the dma-buf fds are imported straight into the DSP service (DSP_IMPORT) — no pixel copy, ~15x faster than the copy-in path on 4K frames.

Rate limiting: the daemon enforces a per-client MPix/s budget (a new client gets a 1-second burst; it then replenishes continuously). Each job is charged src + sum(dst) megapixels, so hot-looping 4K sources (≈8.3 MPix/frame) exhausts the budget within a few jobs and further submissions raise DspError (code == -3, message like “quota: MPix/s budget exhausted”). That error is deliberately NOT silently fallen back to CPU — a switch to CPU is a large latency cliff the app should see. Pace submissions, or crop to a smaller source first.

Usage:

client = DspClient()
small = client.resize_hw(frame.image, 640, 640, fmt="nv12")
tiles = client.multi_crop_hw(frame.image, rects, fmt="nv12")
nv12 = client.convert_hw(frame.image, "nv12", fmt="rgb24")
jpeg = client.encode_jpeg_hw(frame.image, quality=85, fmt="rgb24")
annotated = client.blend_hw(nv12, [(overlay_rgba, 64, 48)])

# zero-copy: keep the frame's dma-bufs and hand them over directly
frame = media.get_frame("main", keep_fd=True)
small = client.resize_hw(frame, 640, 640)
class neoruntime_ipc_sdk.dsp.DspBufferPool(client, width, height, fmt, ids, fds, strides, sizes)[source]

Bases: object

Daemon-allocated dma-buf buffers sharing one geometry.

One wire allocation returns count buffers; every buffer exposes _plane_count(fmt) dma-buf fds (NV12: Y + interleaved-UV). Plane rows may be padded (strides > row bytes); write/read copy row-by-row so padding is preserved. release() returns the buffers to the daemon and closes every fd; closing the client’s UDS releases them too (daemon-side cleanup on disconnect).

__init__(client, width, height, fmt, ids, fds, strides, sizes)[source]
property count: int
buffer_id(index)[source]
write(index, arr)[source]

Copy a numpy array into buffer index (uint8, SDK layout).

nv12: (h*3//2, w) (Y then interleaved UV); rgb24: (h, w, 3); argb: (h, w, 4) (wire byte order [A, R, G, B]); gray8: (h, w).

read(index)[source]

Read buffer index back as a numpy array (SDK layout).

release()[source]

Return all buffers to the daemon (idempotent).

class neoruntime_ipc_sdk.dsp.PendingDspJob(client, reads, job_id, owns, timeout_s, multi=False)[source]

Bases: object

The wait=False return of the *_hw job methods (P2 async).

Wraps one submitted job and owns every buffer it needs — destination pool, imported sources — until the job is consumed:

  • wait() — block, read the destination array(s), release everything. The sync-call equivalent, just later.

  • wait_result() — block but keep the result device-side and the buffers owned: chain buffer_id into DspClient.encode_jpeg_hw() (src_buffer_id=) for a zero read-back encode, then release().

  • done() — poll; timeout_s=0 is a pure non-blocking check.

  • release() — drop the result and free the buffers (idempotent).

A job the daemon refused or a daemon without the async rpcs never produces one of these — refused jobs fall back to CPU and return pixels (the wait=False contract only covers accepted jobs), and the sync fallback inside _submit_job yields job_id=None, meaning the job already ran by construction.

__init__(client, reads, job_id, owns, timeout_s, multi=False)[source]
property buffer_id: int

Daemon id of the (first) destination buffer.

done(timeout_s=0.0)[source]

Poll for completion without consuming the result.

timeout_s=0 maps to the daemon’s non-blocking wait. A job that failed still counts as done — the error surfaces from wait()/wait_result().

wait_result(timeout_s=None)[source]

Block until the job completes, the result staying device-side.

A timed-out job raises but stays pending in the daemon — re-wait with a longer timeout. A failed job raises its error; release the buffers afterwards either way.

wait(timeout_s=None)[source]

Block, read the destination, release the buffers.

release()[source]

Drop the result and release the owned buffers (idempotent).

The daemon-side job is not cancelled: a still-pending job is first reaped with one bounded wait (it executes regardless — the daemon’s single worker runs it either way); a job that outlives that wait lingers in the daemon registry until client disconnect.

class neoruntime_ipc_sdk.dsp.DspClient(sock_path=None, endpoint=None)[source]

Bases: GrpcClient

Hardware resize/crop on the camera-daemon DSP service.

Usage:

dsp = DspClient()
out = dsp.resize_hw(frame.image, 640, 640, fmt="nv12")

The *_hw methods allocate a source and destination buffer, run one job and return the decoded result. For hot loops, pre-allocate pools with alloc_buffers() and pass src_pool/dst_pool (dst_pools for multi-crop) so each call only writes, submits and reads.

Every *_hw method also takes cpu_fallback (default True): when the daemon lacks the DSP surface, array-source calls warn and compute on CPU. Pass cpu_fallback=False to make unavailability raise instead — neoruntime_ipc_sdk.accel does this so its degradation accounting sees the real backend rather than CPU work labeled as hardware. convert_hw() additionally falls back on a job the firmware refused: the pair matrix is device-dependent (hailo15 dsp_convert_format takes RGB<->NV12 and rejects every gray8 pair with HAL_ERR_RESULT), so “the hardware doesn’t do this conversion” is a runtime outcome, not a caller bug.

__init__(sock_path=None, endpoint=None)[source]
close()[source]

Close both transports. The daemon releases our DSP buffers.

alloc_buffers(width, height, fmt='nv12', count=1)[source]

Allocate count daemon-side DSP buffers of one geometry.

resize_hw(src, width, height, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True)[source]

Scale src to (width, height) on the DSP.

src is a numpy array (copied in) or a keep-fd Frame/FrameHandle (imported zero-copy — see the module docstring). wait=False returns a PendingDspJob instead of the array: the job is enqueued without blocking and the buffers stay owned until the pending job consumes them.

crop_hw(src, x, y, width, height, dst_width=None, dst_height=None, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True)[source]

Crop (x, y, w, h) and scale to the destination size.

wait=False returns a PendingDspJob (async submit).

multi_crop_hw(src, rects, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pools=None, cpu_fallback=True, wait=True)[source]

Crop/resize many windows in one job.

rects are (x, y, w, h, dst_width, dst_height). Destination buffers are grouped by geometry (one pool per distinct output size); results come back in rect order. wait=False returns a PendingDspJob whose wait() then yields the list.

convert_hw(src, dst_fmt, fmt=None, priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True)[source]

Convert src to dst_fmt (nv12/rgb24/gray8) on the DSP, keeping the dimensions.

The daemon’s CONVERT contract (P0): source and destination share geometry and differ in format — no rects, exactly one destination buffer. Compose with resize_hw() when you also need scaling, and convert first: NV12 is half the rgb24 bytes, so CONVERT → RESIZE moves less data than the reverse.

Byte order: rgb24 means RGB order on the wire — BGR pixels must be swapped beforehand (or kept on the CPU path via color.bgr_to_nv12); the DSP wire has no BGR variant, so unswapped BGR comes back with R/B-swapped chroma.

Supported pairs are firmware-dependent: on hailo15 only rgb24 <-> nv12 run on the DSP. Every gray8 pair is refused (HAL_ERR_RESULT) — the first refusal warns and falls back to CPU, and this client remembers: later gray8 pairs go straight to the CPU leg, quietly (no repeat warning, last_used_hw=False).

wait=False returns a PendingDspJob (async submit) — only for accepted jobs; a refused pair still returns CPU pixels.

blend_hw(base, overlays, fmt=None, priority='normal', timeout_s=5.0, cpu_fallback=True, wait=True, zero_copy=False)[source]

Composite ARGB32 overlays onto an NV12 base on the DSP (P1).

overlays is a sequence of (rgba, x, y) — an (h, w, 4) uint8 array plus its position on the base; overlays paste 1:1 in order (no scaling, later overlays draw over earlier ones). The blend runs IN PLACE on a daemon pool copy and the annotated NV12 array is returned — the input is never modified.

The base must be NV12 (the vendor op writes NV12 only). Arrays are copied in. Keep-fd Frame/FrameHandle bases are refused by default (zero_copy=False): the import->1:1 RESIZE->BLEND chain has wedged the DSP device-wide until a reboot in the field — twice, under media-heap pressure; a controlled re-test on a healthy heap passed 11/11, so the wedge is state-dependent and the root cause is still open (see docs/proposals/dsp-offload.md P2 record). Pass frame.to_array() — the array path is the proven one. zero_copy=True forces the chain for experiments on future firmware; nothing about it is guaranteed today. Use draw.render_overlay_rgba() to turn detection boxes into a minimal overlay canvas, then blend it here; that keeps the overlay small and the DSP footprint (quota charges base + overlays megapixels) tight.

Overlays smaller than 16x16 (the daemon floor) are padded with fully transparent pixels to 16 — a semantic no-op. Wire byte order is ARGB32 ([A, R, G, B] per pixel); the RGBA->ARGB pack is internal. wait=False submits the blend (and the keep-fd copy leg) without blocking and returns a PendingDspJob.

encode_jpeg_hw(src, quality=85, fmt=None, timeout_s=5.0, src_pool=None, cpu_fallback=True, src_buffer_id=None)[source]

Encode src as one JPEG frame on the camera-daemon (S-3(a)).

Unlike the *_hw job methods this is the daemon’s one-shot EncodeImage RPC: the source is pinned in the DSP registry (imported zero-copy for keep-fd frames, copied into a pool buffer for arrays) and the complete JPEG bytes come back in the response — no destination buffer, no read-back. The daemon owns one standalone encoder keyed by (width, height, format, quality) and recreates it when that key changes, so alternating qualities or geometries re-spins the encoder (first frame after a change pays the pipeline start-up).

Despite the name, the encoder is N-threaded libjpeg on the DSP core behind a GStreamer dispatch — hailo15 has no dedicated JPEG block. The win is central encode + zero-copy input, not raw speed; keep it out of tight per-frame loops that a CPU encode already serves (see docs/proposals/sdk-hardware-routing.md S-3).

quality is 1..100. Inputs are rgb24/nv12 arrays and keep-fd frames (the daemon normalizes RGB through its DSP convert — the encoder pipeline negotiates NV12 only). gray8 arrays up-convert to rgb24 client-side (R=G=B=gray) and ride the same hardware leg; gray8 keep-fd frames raise instead of silently copying — accept the copy yourself with frame.to_array().

src_buffer_id (with src=None) encodes straight from a daemon-side buffer — the zero-copy chain tail: blend_hw(..., wait=False).wait_result().buffer_id lands here and the annotated frame becomes JPEG without a single read-back. There is no CPU fallback on that leg (the client holds no pixels); unavailability raises.

DspClient

class neoruntime_ipc_sdk.DspClient(sock_path=None, endpoint=None)[source]

Bases: GrpcClient

Hardware resize/crop on the camera-daemon DSP service.

Usage:

dsp = DspClient()
out = dsp.resize_hw(frame.image, 640, 640, fmt="nv12")

The *_hw methods allocate a source and destination buffer, run one job and return the decoded result. For hot loops, pre-allocate pools with alloc_buffers() and pass src_pool/dst_pool (dst_pools for multi-crop) so each call only writes, submits and reads.

Every *_hw method also takes cpu_fallback (default True): when the daemon lacks the DSP surface, array-source calls warn and compute on CPU. Pass cpu_fallback=False to make unavailability raise instead — neoruntime_ipc_sdk.accel does this so its degradation accounting sees the real backend rather than CPU work labeled as hardware. convert_hw() additionally falls back on a job the firmware refused: the pair matrix is device-dependent (hailo15 dsp_convert_format takes RGB<->NV12 and rejects every gray8 pair with HAL_ERR_RESULT), so “the hardware doesn’t do this conversion” is a runtime outcome, not a caller bug.

__init__(sock_path=None, endpoint=None)[source]
close()[source]

Close both transports. The daemon releases our DSP buffers.

alloc_buffers(width, height, fmt='nv12', count=1)[source]

Allocate count daemon-side DSP buffers of one geometry.

resize_hw(src, width, height, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True)[source]

Scale src to (width, height) on the DSP.

src is a numpy array (copied in) or a keep-fd Frame/FrameHandle (imported zero-copy — see the module docstring). wait=False returns a PendingDspJob instead of the array: the job is enqueued without blocking and the buffers stay owned until the pending job consumes them.

crop_hw(src, x, y, width, height, dst_width=None, dst_height=None, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True)[source]

Crop (x, y, w, h) and scale to the destination size.

wait=False returns a PendingDspJob (async submit).

multi_crop_hw(src, rects, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pools=None, cpu_fallback=True, wait=True)[source]

Crop/resize many windows in one job.

rects are (x, y, w, h, dst_width, dst_height). Destination buffers are grouped by geometry (one pool per distinct output size); results come back in rect order. wait=False returns a PendingDspJob whose wait() then yields the list.

convert_hw(src, dst_fmt, fmt=None, priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True)[source]

Convert src to dst_fmt (nv12/rgb24/gray8) on the DSP, keeping the dimensions.

The daemon’s CONVERT contract (P0): source and destination share geometry and differ in format — no rects, exactly one destination buffer. Compose with resize_hw() when you also need scaling, and convert first: NV12 is half the rgb24 bytes, so CONVERT → RESIZE moves less data than the reverse.

Byte order: rgb24 means RGB order on the wire — BGR pixels must be swapped beforehand (or kept on the CPU path via color.bgr_to_nv12); the DSP wire has no BGR variant, so unswapped BGR comes back with R/B-swapped chroma.

Supported pairs are firmware-dependent: on hailo15 only rgb24 <-> nv12 run on the DSP. Every gray8 pair is refused (HAL_ERR_RESULT) — the first refusal warns and falls back to CPU, and this client remembers: later gray8 pairs go straight to the CPU leg, quietly (no repeat warning, last_used_hw=False).

wait=False returns a PendingDspJob (async submit) — only for accepted jobs; a refused pair still returns CPU pixels.

blend_hw(base, overlays, fmt=None, priority='normal', timeout_s=5.0, cpu_fallback=True, wait=True, zero_copy=False)[source]

Composite ARGB32 overlays onto an NV12 base on the DSP (P1).

overlays is a sequence of (rgba, x, y) — an (h, w, 4) uint8 array plus its position on the base; overlays paste 1:1 in order (no scaling, later overlays draw over earlier ones). The blend runs IN PLACE on a daemon pool copy and the annotated NV12 array is returned — the input is never modified.

The base must be NV12 (the vendor op writes NV12 only). Arrays are copied in. Keep-fd Frame/FrameHandle bases are refused by default (zero_copy=False): the import->1:1 RESIZE->BLEND chain has wedged the DSP device-wide until a reboot in the field — twice, under media-heap pressure; a controlled re-test on a healthy heap passed 11/11, so the wedge is state-dependent and the root cause is still open (see docs/proposals/dsp-offload.md P2 record). Pass frame.to_array() — the array path is the proven one. zero_copy=True forces the chain for experiments on future firmware; nothing about it is guaranteed today. Use draw.render_overlay_rgba() to turn detection boxes into a minimal overlay canvas, then blend it here; that keeps the overlay small and the DSP footprint (quota charges base + overlays megapixels) tight.

Overlays smaller than 16x16 (the daemon floor) are padded with fully transparent pixels to 16 — a semantic no-op. Wire byte order is ARGB32 ([A, R, G, B] per pixel); the RGBA->ARGB pack is internal. wait=False submits the blend (and the keep-fd copy leg) without blocking and returns a PendingDspJob.

encode_jpeg_hw(src, quality=85, fmt=None, timeout_s=5.0, src_pool=None, cpu_fallback=True, src_buffer_id=None)[source]

Encode src as one JPEG frame on the camera-daemon (S-3(a)).

Unlike the *_hw job methods this is the daemon’s one-shot EncodeImage RPC: the source is pinned in the DSP registry (imported zero-copy for keep-fd frames, copied into a pool buffer for arrays) and the complete JPEG bytes come back in the response — no destination buffer, no read-back. The daemon owns one standalone encoder keyed by (width, height, format, quality) and recreates it when that key changes, so alternating qualities or geometries re-spins the encoder (first frame after a change pays the pipeline start-up).

Despite the name, the encoder is N-threaded libjpeg on the DSP core behind a GStreamer dispatch — hailo15 has no dedicated JPEG block. The win is central encode + zero-copy input, not raw speed; keep it out of tight per-frame loops that a CPU encode already serves (see docs/proposals/sdk-hardware-routing.md S-3).

quality is 1..100. Inputs are rgb24/nv12 arrays and keep-fd frames (the daemon normalizes RGB through its DSP convert — the encoder pipeline negotiates NV12 only). gray8 arrays up-convert to rgb24 client-side (R=G=B=gray) and ride the same hardware leg; gray8 keep-fd frames raise instead of silently copying — accept the copy yourself with frame.to_array().

src_buffer_id (with src=None) encodes straight from a daemon-side buffer — the zero-copy chain tail: blend_hw(..., wait=False).wait_result().buffer_id lands here and the annotated frame becomes JPEG without a single read-back. There is no CPU fallback on that leg (the client holds no pixels); unavailability raises.

DspBufferPool

class neoruntime_ipc_sdk.DspBufferPool(client, width, height, fmt, ids, fds, strides, sizes)[source]

Daemon-allocated dma-buf buffers sharing one geometry.

One wire allocation returns count buffers; every buffer exposes _plane_count(fmt) dma-buf fds (NV12: Y + interleaved-UV). Plane rows may be padded (strides > row bytes); write/read copy row-by-row so padding is preserved. release() returns the buffers to the daemon and closes every fd; closing the client’s UDS releases them too (daemon-side cleanup on disconnect).

__init__(client, width, height, fmt, ids, fds, strides, sizes)[source]
property count: int
buffer_id(index)[source]
write(index, arr)[source]

Copy a numpy array into buffer index (uint8, SDK layout).

nv12: (h*3//2, w) (Y then interleaved UV); rgb24: (h, w, 3); argb: (h, w, 4) (wire byte order [A, R, G, B]); gray8: (h, w).

read(index)[source]

Read buffer index back as a numpy array (SDK layout).

release()[source]

Return all buffers to the daemon (idempotent).

DspError

class neoruntime_ipc_sdk.DspError(message, code=None)[source]

DSP job or buffer error. code mirrors the daemon error codes.

__init__(message, code=None)[source]

PendingDspJob

class neoruntime_ipc_sdk.PendingDspJob(client, reads, job_id, owns, timeout_s, multi=False)[source]

The wait=False return of the *_hw job methods (P2 async).

Wraps one submitted job and owns every buffer it needs — destination pool, imported sources — until the job is consumed:

  • wait() — block, read the destination array(s), release everything. The sync-call equivalent, just later.

  • wait_result() — block but keep the result device-side and the buffers owned: chain buffer_id into DspClient.encode_jpeg_hw() (src_buffer_id=) for a zero read-back encode, then release().

  • done() — poll; timeout_s=0 is a pure non-blocking check.

  • release() — drop the result and free the buffers (idempotent).

A job the daemon refused or a daemon without the async rpcs never produces one of these — refused jobs fall back to CPU and return pixels (the wait=False contract only covers accepted jobs), and the sync fallback inside _submit_job yields job_id=None, meaning the job already ran by construction.

__init__(client, reads, job_id, owns, timeout_s, multi=False)[source]
property buffer_id: int

Daemon id of the (first) destination buffer.

done(timeout_s=0.0)[source]

Poll for completion without consuming the result.

timeout_s=0 maps to the daemon’s non-blocking wait. A job that failed still counts as done — the error surfaces from wait()/wait_result().

wait_result(timeout_s=None)[source]

Block until the job completes, the result staying device-side.

A timed-out job raises but stays pending in the daemon — re-wait with a longer timeout. A failed job raises its error; release the buffers afterwards either way.

wait(timeout_s=None)[source]

Block, read the destination, release the buffers.

release()[source]

Drop the result and release the owned buffers (idempotent).

The daemon-side job is not cancelled: a still-pending job is first reaped with one bounded wait (it executes regardless — the daemon’s single worker runs it either way); a job that outlives that wait lingers in the daemon registry until client disconnect.

Usage Examples

Full-frame resize (model input preprocessing)

from neoruntime_ipc_sdk import FdMediaClient, DspClient, DspError

media = FdMediaClient()
dsp = DspClient()

# keep_fd=True retains the dma-buf fd; DspClient references the
# same buffer zero-copy.
frame = media.get_frame("main", timeout_ms=3000, keep_fd=True)

try:
    # NV12 in -> NV12 out (h + h/2 rows); scaling="stretch" matches
    # cv2.resize semantics.
    small = dsp.resize_hw(frame, 640, 384)
except DspError:
    # When DSP is unavailable and the source is a keep-fd frame the
    # SDK raises instead of silently falling back to CPU — take your
    # own CPU path here.
    small = frame.to_array()

frame.release()  # return the dma-buf early (idempotent, optional)

Batched crops (letterbox scaling)

# rects: (x, y, w, h, dst_w, dst_h) in source pixel coordinates
# (even-aligned); results match the rects order. Ideal for
# "many objects per frame" (e.g. plate tiles) — all crops are
# merged into a single hardware job.
rects = [
    (320, 500, 160, 48, 320, 48),
    (900, 520, 150, 44, 320, 48),
]
tiles = dsp.multi_crop_hw(frame, rects, scaling="letterbox")

Single-region crop

# crop_hw crops and optionally rescales to the target size in one go
tile = dsp.crop_hw(frame, 320, 500, 160, 48, dst_width=320, dst_height=48)

Format conversion (RGB <-> NV12 / grayscale)

# convert_hw swaps the format at identical dimensions (the daemon's
# CONVERT P0 contract): no rects, a single destination buffer;
# dst_fmt is one of "nv12"/"rgb24"/"gray8".
# Byte order: rgb24 on the wire is RGB order — swap BGR pixels
# beforehand (or stay on the CPU path via color.bgr_to_nv12).
nv12 = dsp.convert_hw(rgb, "nv12", fmt="rgb24")
gray = dsp.convert_hw(rgb, "gray8", fmt="rgb24")

# When you also need scaling, CONVERT first, RESIZE second: NV12 is
# about half the rgb24 bytes, so the resize moves half the data.
small = dsp.resize_hw(nv12, 640, 384)

# When the DSP is unavailable the default is a CPU fallback (with a
# UserWarning); pass cpu_fallback=False to raise DspError instead —
# the router uses that for honest degradation accounting.

# The firmware pair matrix is device-dependent: measured on hailo15
# only rgb24 <-> nv12 runs on the DSP — every gray8 pair is
# refused by the firmware (HAL rc=-2801). With the default
# cpu_fallback=True such job rejections also fall back to CPU with a
# warning; last_used_hw=False records the backend actually used.
# After the first refusal the SDK remembers the firmware gap: later
# gray8 array pairs take the CPU leg directly — no warning, no doomed
# submit (keep-fd sources cannot take that leg and keep raising
# honestly).

One-shot JPEG encode (snapshot / thumbnail)

# encode_jpeg_hw is the daemon's one-shot EncodeImage RPC: the source
# buffer is pinned in the DSP registry zero-copy (keep-fd frames import
# their dma-bufs, arrays are copied into a pool buffer) and the
# complete JPEG bytes ride the response — no destination buffer, no
# read-back.
jpeg = dsp.encode_jpeg_hw(frame, quality=85, fmt="nv12")
# array sources default to rgb24: jpeg = dsp.encode_jpeg_hw(rgb, quality=85)

# P2 pass-through leg: src_buffer_id chains the encode onto a pool
# buffer / async job — the result never leaves the device, zero
# read-back (pairs with PendingDspJob.buffer_id under wait=False,
# see "Async jobs" below).
jpeg = dsp.encode_jpeg_hw(None, quality=85, src_buffer_id=job.buffer_id)
# src and src_buffer_id are mutually exclusive; this leg has no
# client pixels to fall back on, so DSP unavailability raises
# DspError directly.

# No "hardware block" despite the name: the encoder is N-threaded
# libjpeg on the DSP core behind a GStreamer dispatch — hailo15 has no
# dedicated JPEG encode block. The win is central encode + zero-copy
# input (app images can drop cv2/PIL), not raw speed; tight per-frame
# loops are still better served by a CPU encode. The daemon reuses one
# encoder keyed by (width, height, format, quality) and recreates it
# when that key changes (the first frame after a change pays the
# pipeline start-up).

Annotation blending (detection boxes onto NV12, dsp-offload P1)

from neoruntime_ipc_sdk import render_overlay_rgba

# blend_hw composites ARGB32 overlays onto an NV12 base, pasted 1:1
# in order (no scaling; later overlays cover earlier ones). The blend
# runs in place on the pool copy — it returns the annotated NV12
# array and never touches the input array.
overlay = np.zeros((64, 96, 4), np.uint8)   # (h, w, 4) RGBA
overlay[..., :3] = (255, 0, 0)
overlay[..., 3] = 255                       # straight alpha
annotated = dsp.blend_hw(nv12, [(overlay, 40, 30)])

# Pair it with render_overlay_rgba for "detection boxes on hardware":
rgba, x0, y0 = render_overlay_rgba(w, h, boxes, labels, scores, colors)
annotated = dsp.blend_hw(nv12, [(rgba, x0, y0)])
# The accel router is the one-call entry:
# router.run("draw_detections", nv12, result)
# Since SDK 0.7.4 draw_detections(nv12, result) rides that route
# itself (RGB input stays on the software raster, keep-fd frames
# raise honestly).

# Contract notes: the base must be NV12 (the vendor op writes NV12
# only). Arrays blend in place on the pool copy — the annotated NV12
# array comes back and the input array is never touched; keep-fd
# frames are **refused by default** (see the firmware-defect note in
# the zero-copy section below); overlays smaller than 16x16 (the
# daemon floor) are padded with fully transparent pixels to 16; the
# hardware ARGB32 memory byte order is [A, R, G, B] and the SDK packs
# it internally; quota is charged on (base + overlays) pixel area —
# keep the canvas minimal (exactly what render_overlay_rgba
# produces). When the DSP is unreachable or the job is rejected,
# behavior matches the other *_hw calls: default warns and falls
# back to CPU (_cpu_blend with identical straight-alpha math);
# cpu_fallback=False raises instead (a keep-fd base has no client
# pixels to fall back on — unavailability always raises there).

Async jobs (submit now, wait later — dsp-offload P2)

# All five job methods (resize/crop/multi_crop/convert/blend) accept
# wait=False: SubmitDspJobAsync returns a job_id immediately and you
# get a PendingDspJob handle. Old daemons without the rpc fall back
# to the synchronous submit transparently — that handle is "born
# done" and wait() costs no extra RPC.
job1 = dsp.resize_hw(frame, 640, 384, wait=False)
job2 = dsp.multi_crop_hw(frame, rects, wait=False)
...  # submission overlaps execution: the single worker thread runs
     # jobs in submission order

small = job1.wait()          # WaitDspJob + pool read-back + release
tiles = job2.wait()          # multi_crop returns a list
if job1.done():              # non-blocking poll (timeout 0); the
    ...                      # completion — including a failure — is
                             # cached after the first report
job1.wait_result()           # wait without reading — result stays
                             # device-side
bid = job1.buffer_id         # chains into encode_jpeg_hw(src_buffer_id=)
job1.release()               # drop the result, return buffers
                             # (idempotent)

# Semantics worth knowing: a wait timeout (rc=-4) keeps the registry
# entry, so you can wait again; a failed job raises DspError from
# the consuming wait (done() polling only reports state); waiting
# after release raises; wait=False does not change the CPU fallback
# story — when a fallback runs you get pixels, not a handle (there
# is no hardware job to wait for).

Zero-copy blend chain (firmware-defect record: refused by default)

# blend_hw with a keep-fd frame raises by default — not a missing
# capability but a safety gate: this chain **has wedged the DSP
# device-wide in the field** (the blend command never returned;
# only a reboot recovers; the xrp driver latches "fatal error,
# reboot required"). The wedge is state-dependent — 2/2 under
# media-heap pressure, 11/11 pass on a healthy heap in a
# controlled re-test; root cause open. Array bases via
# frame.to_array() are the proven, safe path.
try:
    annotated = dsp.blend_hw(frame, [(rgba, x0, y0)])
except DspError:
    annotated = dsp.blend_hw(frame.to_array(), [(rgba, x0, y0)])

# zero_copy=True forces the chain explicitly (import dma-buf -> 1:1
# RESIZE onto a pool -> in-place BLEND; with wait=False the result
# never crosses the socket and encode_jpeg_hw(src_buffer_id=...)
# skips the read-back) — for experiments once firmware is fixed;
# nothing about it is guaranteed today:
job = dsp.blend_hw(frame, [(rgba, x0, y0)],
                   wait=False, zero_copy=True)
job.wait_result()                              # result stays pooled
jpeg = dsp.encode_jpeg_hw(None, src_buffer_id=job.buffer_id)
job.release()                                  # return after encoding

# Ordering note: the RESIZE copy leg always runs synchronously (an
# async job nobody waits would leak its daemon-side registry entry
# and occupy one of the 32 pending slots per connection) — only the
# BLEND compositing leg goes async. Full record:
# docs/proposals/dsp-offload.md, P2 section.

Pre-allocated buffer pool (repeated jobs)

# Repeated jobs with identical geometry can use pooled buffers to
# avoid per-call dma-buf allocation.
pool = dsp.alloc_buffers(640, 384, fmt="nv12", count=4)
small = dsp.resize_hw(frame, 640, 384, dst_pool=pool)

# Read pooled buffer contents
arr = pool.read(0)

# Return the whole pool when done
pool.release()

Context manager

with DspClient() as dsp:
    out = dsp.resize_hw(frame, 416, 416)