The headset sends eye and face in its beacons and pings, and the module saves them as IsEyeSupported and IsFaceSupported in BabbleConfig.json. Offers and pongs report what the module claims and whether VRCFaceTracking must restart. The module only writes the values it claims. Version 4.0.0.
The headset now sends its tracking data as OSC bundles to UDP 6767, from
the same socket as its pings, so the pongs keep the pair open in the PC's
firewall and no inbound rule is needed. Pongs report how many data
datagrams arrived (rx), so the headset can tell when data isn't getting
through. The OSC port 8888 is now for senders on this machine only.
- Prefer IPv4 (A-record) connect with a 4s ConnectTimeout so a .local name opens as fast/deterministically as a raw IP, instead of stalling on an IPv6 link-local past the frame window
- StopCapture disposes the HttpClient (not just cancels) to abort a stuck socket read and close it, freeing the single-connection camera's slot so failover/retry can connect; non-blocking
- Log HTTP status, non-MJPEG content-type, connection failures, and first-frame delivery so silent 'no data' failures become diagnosable
- OpenCvCapture: scope IsOpened() fail-fast to local devices so network streams aren't aborted before the read loop pumps frames
- IpCameraCaptureFactory: accept https MJPEG streams, not just http
- SingleCameraSourceFactory: prefix scheme-less hosts (e.g. openiristracker.local) with http:// so bare endpoints reach a backend
- OpenCVCaptureFactory: claim only an explicit streaming-scheme allowlist instead of any absolute URI
Identify the tracker during enumeration by USB VID 0BB4/PID 0321 or the
"HTC Boot" name, and force the VFT capture backend over the generic
"Normal Camera"/"V4L2 Camera" ones, which otherwise open it un-activated
and mis-decode the image.
BabbleCalibration is a separate .NET (Godot Mono) binary with its own runtime and
no app-local ICU, so under steamrt4 (no libicui18n) its globalization init
FailFast/SIGABRTs at launch, which blocked calibration and training. Launch the
overlay child process with DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1.
Left/right start paths build source but never assign it to VideoSource, and are unreachable: TryStart{Left,Right}IfNotRunning no-op on a SingleCameraSource.
At exit, Avalonia's Linux DBus integration (tray / platform) disconnects on a
background thread and marshals a notification to the already-stopped UI dispatcher,
surfacing as an unhandled TaskCanceledException that aborted the process.
- Add global AppDomain.UnhandledException + TaskScheduler.UnobservedTaskException
handlers (there were none).
- During our shutdown, a benign cancellation now Environment.Exit(0)s instead of
aborting; all other unhandled/unobserved exceptions are logged, not swallowed.
- Complements 76516a9 (Windows exit path); this covers the Linux path.
- App: bounded host dispose then TerminateProcess on Windows exit
- LocalSettings: ForceSave writes synchronously before terminate
- DebounceFunction: add Cancel()
- VFT backend: only probe devices present in enumeration
VRChat / the VRCFT module / DFR can start, stop, or restart at any time relative to
us. Previously a failed connect or bind left the path permanently dead and a send
error was swallowed with no recovery, so any disconnect needed an app restart.
Send (OscSendService):
- Lazily (re)establish the connected UDP socket with backoff, so the consumer can
come up before or after us and output starts flowing on its own.
- Stop trusting Socket.Connected as UDP liveness; a send failure no longer tears the
path down. Transient errors are rate-limited; the socket is rebuilt only when
disposed. On Windows suppress ICMP UDP-connreset so output resumes silently when
the consumer returns.
- Snapshot/lock the socket so a re-target can't race an in-flight send.
Receive (OscRecvService):
- The receive loop owns binding and retries with backoff, so a temporarily-held
in-port (companion app / restart) recovers automatically instead of staying dead.
- Re-targeting closes the socket to wake the in-flight receive; socket access is
serialized; a malformed datagram is isolated so it can't blackout the loop.
VrcftModuleSendService:
- TryParse the stored address with loopback fallback (Parse ran during DI and a bad
stored value crashed host startup); also re-apply on DestinationAddress changes.
ProcessNativeVrcEyeTracking read /leftEyePitch, /leftEyeYaw, /rightEyePitch,
/rightEyeYaw — keys the eye runner never produces (it emits /leftEyeX, /leftEyeY,
/leftEyeLid, ...). The throwing OrderedFloatMap indexer raised KeyNotFoundException
on the first eye frame whenever Use-DFR or VRChat-native eye tracking was enabled;
the exception propagated into ProcessingLoopService.EyeWorker, whose catch calls
StopAllCameras, permanently killing eye tracking until restart.
- Read the produced /leftEyeX//leftEyeY//leftEyeLid keys via non-throwing TryGetValue
- Wrap the native/DFR fan-out so an OSC build error can't reach the worker catch
The device map was cached with ??= and only ever populated on the first start attempt. If that happened with the camera absent (booted with none connected, or clicked start early), the friendly name never resolved to its /dev/video path, so CanConnect rejected every backend and the camera was unopenable until an app restart. Refresh the map once on a lookup miss so a later hot-plug is picked up.
The OneEuroFilter ran upstream of the OSC fan-out, so enabling it also smoothed the DFR (port 9020) and VRChat-native eye streams, adding latency where native eye tracking wants the rawest data. Keep VRCFT filtered; feed the native paths the pre-filter values.
- EyeProcessingPipeline: expose RawEyeResult, the un-smoothed result, geometry-corrected like the returned map but skipping the filter.
- ProcessingLoopService: carry the raw map on the Expressions event.
- ParameterSenderService: DFR and native eye tracking read the raw map; VRCFT keeps the filtered map.
Block the mouth and an eye using the same device (split left+right stays
shared); autostart eyes first so stale configs heal. OpenCvCapture: back
off on failed reads and make StopCapture non-blocking.
The arm64 PatchSkiaSharp target runs `patchelf --add-needed` on libSkiaSharp.so,
but the steamrt4 SDK container has no patchelf (the old ubuntu-arm runner did).
Add it to the CI build prerequisites and the local build image.
Steam Linux Runtime 4 deployment:
- Bundle app-local ICU so the app boots under steamrt4 (no system libicui18n)
- Use the GStreamer-free OpenCV mini-runtime (4.11.0.2) on Linux
- Access V4L2 through libc instead of the unversioned libv4l2 soname (LibV4L2 + VFT)
- Build Linux CI artifacts inside the steamrt4 SDK container
- Exclude the local CUDA trainer bundle (_internal) and fix the MSB3024 copy error
- Fall back across capture backends without an unhandled exception aborting startup
- Add steamrt4 build/run scripts; gitignore .vscode and the trainer _internal bundle
Firmware / serial:
- Surface live espflash progress and a clear failure state instead of always "Done!"
- Stop only serial camera feeds on Refresh (never on tab navigation); release all
firmware serial handles when leaving the tab
- Skip version-unsupported requests (SetPausedRequest on v2) and drop dead-port sessions
- Fix the SerialCameraCapture dispose crash on device unplug
A model hot-swap (training/calibration) replaces the inference service but
reuses the filter, whose per-element state was sized to the first frame. A
model with a different output count then overran those arrays
(IndexOutOfRangeException). Detect a shape change and re-initialize.
Preferring the V4L2 backend only fixes fresh setups; users who already
have "Normal Camera" persisted still get the OpenCV/GStreamer backend,
which can't open /dev/video* without the unbundled v4l2src plugin.
CreateStart now tries every compatible backend in preference order (the
requested one first, then the rest) until one actually produces a frame,
so a dead GStreamer open automatically falls back to LibV4L2. Also drops
the ArgumentNullException-on-no-provider path in favour of a logged null,
and waits on the frame signal instead of busy-polling.
The udev P/Invokes named "libudev.so", the dev-package symlink. Minimal
runtimes like the Steam Linux Runtime (sniper) ship only the versioned
"libudev.so.1", so the load failed there, camera enumeration returned
nothing, friendly names never mapped to /dev/videoN, and starting a camera
threw "No provider ... not found".
Register a DllImportResolver that maps "libudev.so" to whichever soname
exists (libudev.so.1, then .so, then .so.0), so enumeration works under
sniper, scout, and bare distros without the -dev package.
The bundled mini OpenCV runtime captures only through GStreamer, which
needs the v4l2src plugin (gst-plugins-good) to read /dev/video*. That
plugin isn't bundled, so on systems without it the OpenCV "Normal Camera"
backend silently fails. Which backend got used was decided by arbitrary
Directory.GetFiles order, so /dev/video* devices could default to the
broken GStreamer path instead of the dependency-free LibV4L2 one.
- DesktopConnector: deterministically prefer "V4L2 Camera" on Linux.
- OpenCvCapture: when the device fails to open, log an actionable message
(install gst-plugins-good or use V4L2 Camera) and bail instead of
starting a dead capture loop.
- SingleCameraSource.Start: return the backend's real start result so a
failed open fails fast rather than waiting out the 13s frame timeout.
The mini OpenCV runtime has no V4L2 backend, so opening a "/dev/videoN"
path via the string ctor falls back to GStreamer's file source and tries
to read the char device as a media file ("unable to start pipeline").
Resolve the path to an index and open via FromCamera so GStreamer builds
a proper v4l2src pipeline instead.
- Replace the single eye-camera card with a dynamic Cameras list: one card
per active capture, with its own delivered fps, 60s drop count, negotiated
fps, resolution and format
- Handle both a single split eye feed (Eye + Mouth) and two independent
left/right eye feeds (Eye left/right + Mouth)
- Publish the active capture sources from the workers; the sampler reads each
one's counters live, keyed per camera
- Sample every rate against a monotonic Stopwatch clock and read the camera
frame count live, then EWMA-smooth, so delivered fps stops jittering
- Count genuinely lost frames at the capture (overwritten before the pipeline
acquired them) and show the total over a rolling 60s window
- Tidy the Debug page header spacing
- Add a 'UI Render thread' row showing Avalonia's compositor frame rate,
counted via a per-frame RequestCompositionUpdate hook in the view.
- Rename 'UI thread' to 'UI data thread' and the header to 'Performance
metrics'; drop the (responsiveness) and (NNN% = fully saturated) notes.
- Relabel Face -> Mouth (inference rows, pipeline header, worker thread).
- Rename 'CPU hotspots (per thread)' to 'Thread CPU Usage' and
'% of one core' to '% x 1 core'.
The update loop reused one Mat that SetRawMat hands to the consumer, so
the next Read overwrote the buffer the consumer was still using and the
feed stalled. Allocate a fresh Mat per frame and dispose it on a miss.
OpenCV's DSHOW backend can't stream Y800 (raw greyscale) - which some
trackers expose as their default pin - and in fact can't open such
devices at all. Result was an endless "No data received" with the camera
never producing frames on Windows (it works on Linux via GStreamer/MJPEG).
- Add DirectShowModeSelector: enumerate a device's DirectShow modes and
pick the best non-Y800 mode (prefer MJPG, then larger frame, then higher
fps). Generic - no per-camera assumptions.
- OpenCvCapture: apply the selected mode as size -> fps -> FourCC (last),
since the frame rate is what drives the backend onto the MJPG pin when a
camera exposes the same resolution as both raw and compressed. Fall back
to Media Foundation (MSMF) when DSHOW yields no frames; additive, so
cameras already working on DSHOW are unaffected.
- Hand off a fresh Mat per frame in the capture loop. Reusing a single Mat
raced the consumer (concurrent write / use-after-dispose) and crashed
natively (0xC0000005) once frames actually started flowing.
- Log opened backend, negotiated format and a frame probe for diagnostics;
sleep when idle to avoid busy-spinning a CPU core.
- ONNX: cap IntraOpNumThreads=2 to eliminate intra-op thread fan-out CPU.
Eye-only CPU inference on a Ryzen 9 9900X: ~225% -> ~67% of one core
- Add always-on ThreadProfiler + Debug/Performance page: per-thread CPU,
pipeline stage timings, throughput counters
- Settings: add "Split Eye Video Swap" (default unswapped) and opt-in
"Show Debug menu"; move Advanced options to the bottom of the list
- Bundle in-progress expr-dev capture/pipeline work (V4L2 capture, video
sources, OSC recv/send, parameter sender, home/calibration view models)