DSP 硬件加速 API

DSP offload client (SDK-2).

Thin wrapper over the camera-daemon DSP service (platform PLAT-1..6): buffers are allocated on the FD-publisher UDS (/run/aipc/camera.sock, FD_PUB_MSG_DSP_ALLOC/RESP/BUF_RELEASE, dma-buf fds via SCM_RIGHTS) and jobs are submitted through the SubmitDspJob gRPC on camera-control.

Hardware-first with a numpy CPU fallback: on a daemon without the DSP RPC (grpc UNIMPLEMENTED) or with the service not running (error -5) the *_hw helpers compute the result on CPU instead of raising, and client.last_used_hw records which path served the last call.

Caveat (P0 platform contract): a job source must be a daemon-registered dma-buf, so a plain numpy array is copied into one. Zero-copy input IS available for camera frames: pass a Frame received with keep_fd=True` (or its ``.handle) and the dma-buf fds are imported straight into the DSP service (DSP_IMPORT) — no pixel copy, ~15x faster than the copy-in path on 4K frames.

Rate limiting: the daemon enforces a per-client MPix/s budget (a new client gets a 1-second burst; it then replenishes continuously). Each job is charged src + sum(dst) megapixels, so hot-looping 4K sources (≈8.3 MPix/frame) exhausts the budget within a few jobs and further submissions raise DspError (code == -3, message like “quota: MPix/s budget exhausted”). That error is deliberately NOT silently fallen back to CPU — a switch to CPU is a large latency cliff the app should see. Pace submissions, or crop to a smaller source first.

Usage:

client = DspClient()
small = client.resize_hw(frame.image, 640, 640, fmt="nv12")
tiles = client.multi_crop_hw(frame.image, rects, fmt="nv12")
nv12 = client.convert_hw(frame.image, "nv12", fmt="rgb24")
jpeg = client.encode_jpeg_hw(frame.image, quality=85, fmt="rgb24")
annotated = client.blend_hw(nv12, [(overlay_rgba, 64, 48)])

# zero-copy: keep the frame's dma-bufs and hand them over directly
frame = media.get_frame("main", keep_fd=True)
small = client.resize_hw(frame, 640, 640)

# device-resident chain: results stay on the daemon between calls,
# no read-back until the tail (P2-2)
ref = client.resize_hw(frame, 640, 640, out="ref")
jpeg = client.encode_jpeg_hw(ref, quality=85)
ref.release()

class neoruntime_ipc_sdk.dsp.DspBufferPool(client, width, height, fmt, ids, fds, strides, sizes)[source]

Bases: object

Daemon-allocated dma-buf buffers sharing one geometry.

One wire allocation returns count buffers; every buffer exposes _plane_count(fmt) dma-buf fds (NV12: Y + interleaved-UV). Plane rows may be padded (strides > row bytes); write/read copy row-by-row so padding is preserved. release() returns the buffers to the daemon and closes every fd; closing the client’s UDS releases them too (daemon-side cleanup on disconnect).

__init__(client, width, height, fmt, ids, fds, strides, sizes)[source]

property count: int

buffer_id(index)[source]

write(index, arr)[source]

Copy a numpy array into buffer index (uint8, SDK layout).

nv12: (h*3//2, w) (Y then interleaved UV); rgb24: (h, w, 3); argb: (h, w, 4) (wire byte order [A, R, G, B]); gray8: (h, w).

read(index)[source]

Read buffer index back as a numpy array (SDK layout).

release()[source]

Return all buffers to the daemon (idempotent).

class neoruntime_ipc_sdk.dsp.SharedImportPool(client, width, height, ids, maps, fds)[source]

Bases: object

A round-robin ring of shared ARGB32 imports the daemon can bake.

The deployed HAL refuses ARGB32 dma-buf pool allocation (wire OOM), but its bake path blends ARGB imports fine: the daemon maps the memfd we pass over SCM_RIGHTS as USERPTR planes, so both processes share the same pages. write() therefore lands directly in the memory the daemon’s blend reads — no dma-buf, no copy through the wire.

Lease rules mirror DspBufferPool: slots recycle round-robin, the caller paces writes, and a slot must not be rewritten while its frame may still sit in a daemon-side queue (queue cap 3 ⇒ ring depth >= 4 for faster-than-bake publishing).

Not per-publish import/release: client-streaming has no per-frame ack, so a release could race the daemon’s pin. Not hold-until-RPC-end either: the daemon caps imports at 64 per client. A small persistent ring sidesteps both.

__init__(client, width, height, ids, maps, fds)[source]

property count: int: Number of ring slots.

property strides: tuple[int, ...]: Per-plane byte strides — ARGB32 is one width * 4 plane.

buffer_id(index)[source]

The daemon registry id of ring slot index.

write(index, arr)[source]

Copy one ARGB32 frame into ring slot index’s shared pages.

release()[source]

Return every import and close the shared pages (idempotent).

class neoruntime_ipc_sdk.dsp.PendingDspJob(client, reads, job_id, owns, timeout_s, multi=False)[source]

Bases: object

The wait=False return of the *_hw job methods (P2 async).

Wraps one submitted job and owns every buffer it needs — destination pool, imported sources — until the job is consumed:

wait() — block, read the destination array(s), release everything. The sync-call equivalent, just later.
wait_result() — block but keep the result device-side and the buffers owned: chain buffer_id into DspClient.encode_jpeg_hw() (src_buffer_id=) for a zero read-back encode, then release().
done() — poll; timeout_s=0 is a pure non-blocking check.
release() — drop the result and free the buffers (idempotent).

A job the daemon refused or a daemon without the async rpcs never produces one of these — refused jobs fall back to CPU and return pixels (the wait=False contract only covers accepted jobs), and the sync fallback inside _submit_job yields job_id=None, meaning the job already ran by construction.

__init__(client, reads, job_id, owns, timeout_s, multi=False)[source]

property buffer_id: int: Daemon id of the (first) destination buffer.

done(timeout_s=0.0)[source]

Poll for completion without consuming the result.

timeout_s=0 maps to the daemon’s non-blocking wait. A job that failed still counts as done — the error surfaces from wait()/wait_result().

wait_result(timeout_s=None)[source]

Block until the job completes, the result staying device-side.

A timed-out job raises but stays pending in the daemon — re-wait with a longer timeout. A failed job raises its error; release the buffers afterwards either way.

wait(timeout_s=None)[source]

Block, read the destination, release the buffers.

release()[source]

Drop the result and release the owned buffers (idempotent).

The daemon-side job is not cancelled: a still-pending job is first reaped with one bounded wait (it executes regardless — the daemon’s single worker runs it either way); a job that outlives that wait lingers in the daemon registry until client disconnect.

class neoruntime_ipc_sdk.dsp.DspBufferRef(client, pool, index, owns)[source]

Bases: object

The out="ref" return of the sync *_hw job methods (P2-2).

Wraps one completed job’s destination buffer — device-side, no read-back — plus everything the call owned to produce it (imported sources, temp pools):

buffer_id/width/height/fmt — the daemon buffer id and its geometry; chain the id into another *_hw call or encode_jpeg_hw without touching the pixels.
read() — read the array back on demand (exactly what the out=None default returns inline).
release() — return the owned buffers to the daemon (idempotent). When the destination pool was caller-supplied (dst_pool=) the pool is not owned and stays valid — but the ref is dead either way; re-read through the pool if you need it.

A ref is also a valid job source: passing it as src uses its daemon id directly (no re-import, no copy-in). That borrows it — the source call neither consumes nor keeps the ref alive; keep it alive until the consuming job completes, then release() it. There is no GC safety net (same contract as PendingDspJob): rebinding a ref away without release() leaks its daemon-side pixel budget until the client disconnects — chain users keep every intermediate ref and release it, or use one ref per statement.

Context manager support releases on exit. Device-side lifetime: like every client-owned buffer, the daemon reclaims it when the client’s UDS closes — a leaked ref costs its slot until then. DspClient.close() (explicit, or the accel router dropping the process-resident client after a failed call) also poisons the pool, so a ref outliving its client raises on next use instead of submitting the reclaimed id.

__init__(client, pool, index, owns)[source]

property buffer_id: int: Daemon id of the wrapped buffer.

property width: int

property height: int

property fmt: str

property released: bool

read()[source]

Read the buffer back as a numpy array (SDK layout).

release()[source]

Release the owned buffers (idempotent).

Frees what the producing call allocated — temp pools and imports; a caller-supplied dst_pool is left alone. The job itself already completed (out="ref" is the sync path), so there is nothing to reap, unlike PendingDspJob.

class neoruntime_ipc_sdk.dsp.DspClient(sock_path=None, endpoint=None)[source]

Bases: GrpcClient

Hardware resize/crop on the camera-daemon DSP service.

Usage:

dsp = DspClient()
out = dsp.resize_hw(frame.image, 640, 640, fmt="nv12")

The *_hw methods allocate a source and destination buffer, run one job and return the decoded result. For hot loops, pre-allocate pools with alloc_buffers() and pass src_pool/dst_pool (dst_pools for multi-crop) so each call only writes, submits and reads.

Every *_hw method also takes cpu_fallback (default True): when the daemon lacks the DSP surface, array-source calls warn and compute on CPU. Pass cpu_fallback=False to make unavailability raise instead — neoruntime_ipc_sdk.accel does this so its degradation accounting sees the real backend rather than CPU work labeled as hardware. convert_hw() additionally falls back on a job the firmware refused: the pair matrix is device-dependent (hailo15 dsp_convert_format takes RGB<->NV12 and rejects every gray8 pair with HAL_ERR_RESULT), so “the hardware doesn’t do this conversion” is a runtime outcome, not a caller bug.

__init__(sock_path=None, endpoint=None)[source]

has_live_pools()[source]

True while a pool this client allocated is still usable.

The accel router’s retirement drain polls this before closing a failed resident: pools die with their refs (the WeakSet) or turn stale on release(), so flipping to False means no ref can still be reading through this client.

close()[source]

Close both transports. The daemon releases our DSP buffers.

Still-live pools this client allocated are poisoned first: the daemon reclaims their buffers when the UDS drops, so a held DspBufferRef would dangle — after close, its (and the pool’s) next use raises instead of submitting reclaimed ids.

alloc_buffers(width, height, fmt='nv12', count=1)[source]

Allocate count daemon-side DSP buffers of one geometry.

import_shared_buffers(width, height, fmt='argb', count=2, timeout_s=5.0)[source]

Import count shared ARGB32 buffers as a writable ring.

The deployed HAL refuses ARGB32 dma-buf pool allocation, so alpha-blend overlays ride memfd imports instead: each slot is a memfd the daemon maps (SCM_RIGHTS dup, MAP_SHARED both sides) and bakes straight from. Writing the returned pool puts pixels in the exact pages the daemon’s blend reads. See SharedImportPool for the lease contract and why the ring is persistent.

import_frame(handle)[source]

Import a frame’s dma-bufs into the daemon buffer registry (DSP_IMPORT).

The daemon dups the fds, so the returned id outlives the FrameHandle; it lives in the same registry namespace as pool buffer ids and stays valid until release_buffer() (or the daemon’s lease watchdog). This is the zero-copy entry point for InferenceClient.infer(frame, ...).

release_buffer(buffer_id)[source]

Free a registry id (import id or pool buffer) with DSP_BUF_RELEASE.

resize_hw(src, width, height, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True, out=None)[source]

Scale src to (width, height) on the DSP.

src is a numpy array (copied in), a keep-fd Frame/FrameHandle (imported zero-copy — see the module docstring), or a DspBufferRef (its daemon buffer used directly — no re-import). wait=False returns a PendingDspJob instead of the array: the job is enqueued without blocking and the buffers stay owned until the pending job consumes them. out="ref" (sync path only) skips the read-back and returns a DspBufferRef — the resized pixels stay device-side for the next hw call; ref.read() brings them back on demand.

crop_hw(src, x, y, width, height, dst_width=None, dst_height=None, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True, out=None)[source]

Crop (x, y, w, h) and scale to the destination size.

wait=False returns a PendingDspJob (async submit). out="ref" (sync path only) returns a DspBufferRef instead of reading the pixels back — see resize_hw().

multi_crop_hw(src, rects, fmt=None, interpolation='bilinear', scaling='stretch', priority='normal', timeout_s=5.0, src_pool=None, dst_pools=None, cpu_fallback=True, wait=True)[source]

Crop/resize many windows in one job.

rects are (x, y, w, h, dst_width, dst_height). Destination buffers are grouped by geometry (one pool per distinct output size); results come back in rect order. wait=False returns a PendingDspJob whose wait() then yields the list.

convert_hw(src, dst_fmt, fmt=None, priority='normal', timeout_s=5.0, src_pool=None, dst_pool=None, cpu_fallback=True, wait=True, out=None)[source]

Convert src to dst_fmt (nv12/rgb24/gray8) on the DSP, keeping the dimensions.

The daemon’s CONVERT contract (P0): source and destination share geometry and differ in format — no rects, exactly one destination buffer. Compose with resize_hw() when you also need scaling, and convert first: NV12 is half the rgb24 bytes, so CONVERT → RESIZE moves less data than the reverse.

Byte order: rgb24 means RGB order on the wire — BGR pixels must be swapped beforehand (or kept on the CPU path via color.bgr_to_nv12); the DSP wire has no BGR variant, so unswapped BGR comes back with R/B-swapped chroma.

Supported pairs are firmware-dependent: on hailo15 only rgb24 <-> nv12 run on the DSP. Every gray8 pair is refused (HAL_ERR_RESULT) — the first refusal warns and falls back to CPU, and this client remembers: later gray8 pairs go straight to the CPU leg, quietly (no repeat warning, last_used_hw=False).

wait=False returns a PendingDspJob (async submit) — only for accepted jobs; a refused pair still returns CPU pixels. out="ref" (sync path only) returns a DspBufferRef instead of reading the pixels back — see resize_hw().

blend_hw(base, overlays, fmt=None, priority='normal', timeout_s=5.0, cpu_fallback=True, wait=True, zero_copy=False, out=None, dst_slot=0)[source]

Composite ARGB32 overlays onto an NV12 base on the DSP (P1).

overlays is a sequence of (rgba, x, y) — an (h, w, 4) uint8 array plus its position on the base; overlays paste 1:1 in order (no scaling, later overlays draw over earlier ones). The blend runs IN PLACE on a daemon pool copy and the annotated NV12 array is returned — the input is never modified.

The base must be NV12 (the vendor op writes NV12 only). Arrays are copied in. Device-side bases — keep-fd Frame/FrameHandle (an import) or a DspBufferRef (its daemon id, borrowed: keep the ref alive until the call returns, like any ref source) — are refused by default (zero_copy=False): the import->1:1 RESIZE->BLEND chain has wedged the DSP device-wide until a reboot in the field — twice, under media-heap pressure; a controlled re-test on a healthy heap passed 11/11 and a 4000-run soak saw no wedge, so the risk is state-dependent with the root cause still open (see docs/proposals/dsp-offload.md P2 record). Pass frame.to_array()/ref.read() — the array path is the proven one. zero_copy=True forces the device-side chain (the ref leg skips the import entirely: one in-daemon RESIZE copy, then the blend); nothing about it is guaranteed today. Use draw.render_overlay_rgba() to turn detection boxes into a minimal overlay canvas, then blend it here; that keeps the overlay small and the DSP footprint (quota charges base + overlays megapixels) tight.

Overlays smaller than 16x16 (the daemon floor) are padded with fully transparent pixels to 16, and odd overlay heights gain one transparent bottom row (the DSP rejects odd overlay heights outright, -2801 — device-ruled: odd width/x/y pass, only height does not). Both pads are semantic no-ops, but they grow the rect: bounds are enforced on the padded geometry. Wire byte order is ARGB32 ([A, R, G, B] per pixel); the RGBA->ARGB pack is internal. wait=False submits the blend (and the keep-fd copy leg) without blocking and returns a PendingDspJob.

out= picks the destination (sync path only):

None (default) — read the annotated pixels back as an ndarray.
"ref" — skip the read-back; return a DspBufferRef wrapping the annotated buffer (chain its buffer_id into encode_jpeg_hw or another *_hw call, read() on demand).
a DspBufferPool — blend straight into the caller’s pool at dst_slot and return None: no allocation, no read-back, nothing released (the pool stays the caller’s). The pool must match the base geometry and be NV12 (blend keeps dims). This is the zero-copy publish tail — a FramePublisher pool slot filled here is pushed with push_slot — and with an array base the CPU fallback (if taken) writes the slot too, so the destination contract holds on every path.

encode_jpeg_hw(src, quality=85, fmt=None, timeout_s=5.0, src_pool=None, cpu_fallback=True, src_buffer_id=None)[source]

Encode src as one JPEG frame on the camera-daemon (S-3(a)).

Unlike the *_hw job methods this is the daemon’s one-shot EncodeImage RPC: the source is pinned in the DSP registry (imported zero-copy for keep-fd frames, copied into a pool buffer for arrays) and the complete JPEG bytes come back in the response — no destination buffer, no read-back. The daemon owns one standalone encoder keyed by (width, height, format, quality) and recreates it when that key changes, so alternating qualities or geometries re-spins the encoder (first frame after a change pays the pipeline start-up).

Despite the name, the encoder is N-threaded libjpeg on the DSP core behind a GStreamer dispatch — hailo15 has no dedicated JPEG block. The win is central encode + zero-copy input, not raw speed; keep it out of tight per-frame loops that a CPU encode already serves (see docs/proposals/sdk-hardware-routing.md S-3).

quality is 1..100. Inputs are rgb24/nv12 arrays and keep-fd frames (the daemon normalizes RGB through its DSP convert — the encoder pipeline negotiates NV12 only). gray8 arrays up-convert to rgb24 client-side (R=G=B=gray) and ride the same hardware leg; gray8 keep-fd frames raise instead of silently copying — accept the copy yourself with frame.to_array(). A DspBufferRef encodes straight from its daemon buffer (the same zero-copy leg as src_buffer_id).

src_buffer_id (with src=None) encodes straight from a daemon-side buffer — the zero-copy chain tail: blend_hw(..., wait=False).wait_result().buffer_id lands here and the annotated frame becomes JPEG without a single read-back. There is no CPU fallback on that leg (the client holds no pixels); unavailability raises.

DspClient

class neoruntime_ipc_sdk.DspClient(sock_path=None, endpoint=None)[source]

Bases: GrpcClient

Hardware resize/crop on the camera-daemon DSP service.

Usage:

dsp = DspClient()
out = dsp.resize_hw(frame.image, 640, 640, fmt="nv12")