Commit Graph
662 Commits
Author SHA1 Message Date
dfgHiatus 902acf816f feat: Add MetadataUpdatePacket support vrcft-module-4.0.0 2026-10-27 19:01:54 -07:00
Rames The GenericandClaude Opus 5.5 d5f78cd1e4 Point Babble VRCFT module DownloadUrl at the 4.0.0 release
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-27 19:01:53 -07:00
Rames The GenericandClaude Opus 5.5 bb77b9f347 Set Babble VRCFT module version to 4.0.0
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-27 19:01:53 -07:00
Rames The Generic 06ada15000 BFR Link: headset sets eye and face claims, module reports its mode
The headset sends eye and face in its beacons and pings, and the module saves them as IsEyeSupported and IsFaceSupported in BabbleConfig.json. Offers and pongs report what the module claims and whether VRCFaceTracking must restart. The module only writes the values it claims. Version 4.0.0.
2026-10-27 19:01:53 -07:00
Rames The Generic e8eca36674 BFR Link: receive headset data on the link port
The headset now sends its tracking data as OSC bundles to UDP 6767, from
the same socket as its pings, so the pongs keep the pair open in the PC's
firewall and no inbound rule is needed. Pongs report how many data
datagrams arrived (rx), so the headset can tell when data isn't getting
through. The OSC port 8888 is now for senders on this machine only.
2026-10-27 19:01:52 -07:00
Rames The Generic 9adcd5e6b5 Initial protocol support for BFR 2026-10-27 19:01:52 -07:00
dan2wik 729933f070 IpCameraCapture: IPv4-preferring connect, forceful teardown, error logging
- Prefer IPv4 (A-record) connect with a 4s ConnectTimeout so a .local name opens as fast/deterministically as a raw IP, instead of stalling on an IPv6 link-local past the frame window
- StopCapture disposes the HttpClient (not just cancels) to abort a stuck socket read and close it, freeing the single-connection camera's slot so failover/retry can connect; non-blocking
- Log HTTP status, non-MJPEG content-type, connection failures, and first-frame delivery so silent 'no data' failures become diagnosable
2026-10-27 19:01:51 -07:00
dan2wik 11545dd1f4 Drive titlebar version from informational version; allow CI override
- MainWindow: read AssemblyInformationalVersion (keeps -prerelease suffix), strip +<git-sha>
- Baballonia.csproj: add Version default, disable source-revision suffix
- build.yml: pass -p:Version from the workflow version-number input when set
2026-10-27 19:01:51 -07:00
dan2wik 3b5849cc89 Fix streaming-camera regression; tighten capture URL routing
- OpenCvCapture: scope IsOpened() fail-fast to local devices so network streams aren't aborted before the read loop pumps frames
- IpCameraCaptureFactory: accept https MJPEG streams, not just http
- SingleCameraSourceFactory: prefix scheme-less hosts (e.g. openiristracker.local) with http:// so bare endpoints reach a backend
- OpenCVCaptureFactory: claim only an explicit streaming-scheme allowlist instead of any absolute URI
2026-10-27 19:01:51 -07:00
dan2wik 41603f3863 DefaultInferenceRunner: throw when output size matches no known layout 2026-10-27 19:01:50 -07:00
dan2wik 092f20b9fa DefaultInferenceRunner: demote model-expr init log to Debug 2026-10-27 19:01:50 -07:00
dan2wik 4198129ebe OneEuroFilter: copy spans via CopyTo in zero-dt branch 2026-10-27 19:01:50 -07:00
dan2wik 28c579af36 OneEuroFilter: use MathF.Tau instead of local TwoPi const 2026-10-27 19:01:49 -07:00
dan2wik 05de6e8c1c Prefer the VFT backend for identified Vive Facial Trackers
Identify the tracker during enumeration by USB VID 0BB4/PID 0321 or the
"HTC Boot" name, and force the VFT capture backend over the generic
"Normal Camera"/"V4L2 Camera" ones, which otherwise open it un-activated
and mis-decode the image.
2026-10-27 19:01:49 -07:00
dan2wik 4b44b80213 Run the calibration overlay invariant so it boots under Steam Runtime 4
BabbleCalibration is a separate .NET (Godot Mono) binary with its own runtime and
no app-local ICU, so under steamrt4 (no libicui18n) its globalization init
FailFast/SIGABRTs at launch, which blocked calibration and training. Launch the
overlay child process with DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1.
2026-10-27 19:01:49 -07:00
dan2wik 6a0512cba6 Renaming linux workaround key to avoid stale translations 2026-10-27 19:01:48 -07:00
dan2wik 2e48440c3b Updated and improved linux workaround instructions. 2026-10-27 19:01:48 -07:00
dan2wik 46d5d5e580 Flag flawed/unused SingleCameraSource branches in EyePipelineManager
Left/right start paths build source but never assign it to VideoSource, and are unreachable: TryStart{Left,Right}IfNotRunning no-op on a SingleCameraSource.
2026-10-27 19:01:47 -07:00
dan2wik dbbfc8ee74 Mount the BabbleTrainer symlink target into the steamrt4 build container 2026-10-27 19:01:47 -07:00
dan2wik 40c5a31fd4 Fix SteamVR autostart toggle off path and crash
- Apply user toggles reliably instead of swallowing them during load
- Use a short-lived OpenVR session so toggling after SteamVR closes can't crash
2026-10-27 19:01:47 -07:00
dan2wik afa8c6c5c0 Fix build warnings and the broken SteamVR autostart toggle
- Clear NU1510/CA2017/CS0105/CS0649/MSB3568 warnings
- Wire up OpenVRService so the autostart toggle works
2026-10-27 19:01:46 -07:00
dan2wik 107cc5e681 Bump Tmds.DBus.Protocol to 0.21.3 to fix GHSA-xrw6-gwf8-vvr9 2026-10-27 19:01:46 -07:00
dan2wik b5642cd80f Exit cleanly on the Linux DBus teardown crash at shutdown
At exit, Avalonia's Linux DBus integration (tray / platform) disconnects on a
background thread and marshals a notification to the already-stopped UI dispatcher,
surfacing as an unhandled TaskCanceledException that aborted the process.

- Add global AppDomain.UnhandledException + TaskScheduler.UnobservedTaskException
  handlers (there were none).
- During our shutdown, a benign cancellation now Environment.Exit(0)s instead of
  aborting; all other unhandled/unobserved exceptions are logged, not swallowed.
- Complements 76516a9 (Windows exit path); this covers the Linux path.
2026-10-27 19:01:46 -07:00
dan2wik 080d3fb481 Fix Windows exit hang from wedged camera capture thread
- App: bounded host dispose then TerminateProcess on Windows exit
- LocalSettings: ForceSave writes synchronously before terminate
- DebounceFunction: add Cancel()
- VFT backend: only probe devices present in enumeration
2026-10-27 19:01:45 -07:00
dan2wik 462e808069 Make OSC send/receive self-healing across consumer restarts
VRChat / the VRCFT module / DFR can start, stop, or restart at any time relative to
us. Previously a failed connect or bind left the path permanently dead and a send
error was swallowed with no recovery, so any disconnect needed an app restart.

Send (OscSendService):
- Lazily (re)establish the connected UDP socket with backoff, so the consumer can
  come up before or after us and output starts flowing on its own.
- Stop trusting Socket.Connected as UDP liveness; a send failure no longer tears the
  path down. Transient errors are rate-limited; the socket is rebuilt only when
  disposed. On Windows suppress ICMP UDP-connreset so output resumes silently when
  the consumer returns.
- Snapshot/lock the socket so a re-target can't race an in-flight send.

Receive (OscRecvService):
- The receive loop owns binding and retries with backoff, so a temporarily-held
  in-port (companion app / restart) recovers automatically instead of staying dead.
- Re-targeting closes the socket to wake the in-flight receive; socket access is
  serialized; a malformed datagram is isolated so it can't blackout the loop.

VrcftModuleSendService:
- TryParse the stored address with loopback fallback (Parse ran during DI and a bad
  stored value crashed host startup); also re-apply on DestinationAddress changes.
2026-10-27 19:01:45 -07:00
dan2wik f2f1723a73 Fix DFR/native eye OSC crash that stopped all eye cameras
ProcessNativeVrcEyeTracking read /leftEyePitch, /leftEyeYaw, /rightEyePitch,
/rightEyeYaw — keys the eye runner never produces (it emits /leftEyeX, /leftEyeY,
/leftEyeLid, ...). The throwing OrderedFloatMap indexer raised KeyNotFoundException
on the first eye frame whenever Use-DFR or VRChat-native eye tracking was enabled;
the exception propagated into ProcessingLoopService.EyeWorker, whose catch calls
StopAllCameras, permanently killing eye tracking until restart.

- Read the produced /leftEyeX//leftEyeY//leftEyeLid keys via non-throwing TryGetValue
- Wrap the native/DFR fan-out so an OSC build error can't reach the worker catch
2026-10-27 19:01:45 -07:00
dan2wik 8e9e0ff901 Re-enumerate cameras on a friendly-name miss
The device map was cached with ??= and only ever populated on the first start attempt. If that happened with the camera absent (booted with none connected, or clicked start early), the friendly name never resolved to its /dev/video path, so CanConnect rejected every backend and the camera was unopenable until an app restart. Refresh the map once on a lookup miss so a later hot-plug is picked up.
2026-10-27 19:01:44 -07:00
dan2wik 80473d6c6e Send raw, unfiltered eye data to the DFR/native OSC output
The OneEuroFilter ran upstream of the OSC fan-out, so enabling it also smoothed the DFR (port 9020) and VRChat-native eye streams, adding latency where native eye tracking wants the rawest data. Keep VRCFT filtered; feed the native paths the pre-filter values.

- EyeProcessingPipeline: expose RawEyeResult, the un-smoothed result, geometry-corrected like the returned map but skipping the filter.
- ProcessingLoopService: carry the raw map on the Expressions event.
- ParameterSenderService: DFR and native eye tracking read the raw map; VRCFT keeps the filtered map.
2026-10-27 19:01:44 -07:00
dan2wik 0bbf7e8d1f Prevent eye/mouth sharing one camera; fix capture wedging
Block the mouth and an eye using the same device (split left+right stays
shared); autostart eyes first so stale configs heal. OpenCvCapture: back
off on failed reads and make StopCapture non-blocking.
2026-10-27 19:01:43 -07:00
dan2wik a74f60f5bc Show Linux CI jobs as steamrt4-x64 / steamrt4-arm 2026-10-27 19:01:43 -07:00
dan2wik 9ebb821e5d Install patchelf in the steamrt4 build container for the arm64 SkiaSharp fixup
The arm64 PatchSkiaSharp target runs `patchelf --add-needed` on libSkiaSharp.so,
but the steamrt4 SDK container has no patchelf (the old ubuntu-arm runner did).
Add it to the CI build prerequisites and the local build image.
2026-10-27 19:01:43 -07:00
dan2wik 460e36c549 Run under Steam Runtime 4 and harden firmware/serial handling
Steam Linux Runtime 4 deployment:
- Bundle app-local ICU so the app boots under steamrt4 (no system libicui18n)
- Use the GStreamer-free OpenCV mini-runtime (4.11.0.2) on Linux
- Access V4L2 through libc instead of the unversioned libv4l2 soname (LibV4L2 + VFT)
- Build Linux CI artifacts inside the steamrt4 SDK container
- Exclude the local CUDA trainer bundle (_internal) and fix the MSB3024 copy error
- Fall back across capture backends without an unhandled exception aborting startup
- Add steamrt4 build/run scripts; gitignore .vscode and the trainer _internal bundle

Firmware / serial:
- Surface live espflash progress and a clear failure state instead of always "Done!"
- Stop only serial camera feeds on Refresh (never on tab navigation); release all
  firmware serial handles when leaving the tab
- Skip version-unsupported requests (SetPausedRequest on v2) and drop dead-port sessions
- Fix the SerialCameraCapture dispose crash on device unplug
2026-10-27 19:01:42 -07:00
dan2wik 7741121046 Re-init OneEuroFilter when the model output shape changes
A model hot-swap (training/calibration) replaces the inference service but
reuses the filter, whose per-element state was sized to the first frame. A
model with a different output count then overran those arrays
(IndexOutOfRangeException). Detect a shape change and re-initialize.
2026-10-27 19:01:42 -07:00
dan2wik 636a49c3ee Fall back to the next capture backend when one can't deliver frames
Preferring the V4L2 backend only fixes fresh setups; users who already
have "Normal Camera" persisted still get the OpenCV/GStreamer backend,
which can't open /dev/video* without the unbundled v4l2src plugin.

CreateStart now tries every compatible backend in preference order (the
requested one first, then the rest) until one actually produces a frame,
so a dead GStreamer open automatically falls back to LibV4L2. Also drops
the ArgumentNullException-on-no-provider path in favour of a logged null,
and waits on the frame signal instead of busy-polling.
2026-10-27 19:01:42 -07:00
dan2wik 9fba4e67d4 Load libudev by versioned soname so enumeration works under Steam runtime
The udev P/Invokes named "libudev.so", the dev-package symlink. Minimal
runtimes like the Steam Linux Runtime (sniper) ship only the versioned
"libudev.so.1", so the load failed there, camera enumeration returned
nothing, friendly names never mapped to /dev/videoN, and starting a camera
threw "No provider ... not found".

Register a DllImportResolver that maps "libudev.so" to whichever soname
exists (libudev.so.1, then .so, then .so.0), so enumeration works under
sniper, scout, and bare distros without the -dev package.
2026-10-27 19:01:41 -07:00
dan2wik 4b972e9fb6 Prefer V4L2 backend on Linux and fail fast when GStreamer can't open
The bundled mini OpenCV runtime captures only through GStreamer, which
needs the v4l2src plugin (gst-plugins-good) to read /dev/video*. That
plugin isn't bundled, so on systems without it the OpenCV "Normal Camera"
backend silently fails. Which backend got used was decided by arbitrary
Directory.GetFiles order, so /dev/video* devices could default to the
broken GStreamer path instead of the dependency-free LibV4L2 one.

- DesktopConnector: deterministically prefer "V4L2 Camera" on Linux.
- OpenCvCapture: when the device fails to open, log an actionable message
  (install gst-plugins-good or use V4L2 Camera) and bail instead of
  starting a dead capture loop.
- SingleCameraSource.Start: return the backend's real start result so a
  failed open fails fast rather than waiting out the 13s frame timeout.
2026-10-27 19:01:41 -07:00
dan2wik 226976c28a Fix Linux capture failing to open /dev/videoN paths
The mini OpenCV runtime has no V4L2 backend, so opening a "/dev/videoN"
path via the string ctor falls back to GStreamer's file source and tries
to read the char device as a media file ("unable to start pipeline").
Resolve the path to an index and open via FromCamera so GStreamer builds
a proper v4l2src pipeline instead.
2026-10-27 19:01:41 -07:00
dan2wik b016748179 Show per-camera debug stats for up to 3 cameras
- Replace the single eye-camera card with a dynamic Cameras list: one card
  per active capture, with its own delivered fps, 60s drop count, negotiated
  fps, resolution and format
- Handle both a single split eye feed (Eye + Mouth) and two independent
  left/right eye feeds (Eye left/right + Mouth)
- Publish the active capture sources from the workers; the sampler reads each
  one's counters live, keyed per camera
2026-10-27 19:01:40 -07:00
dan2wik 72cb009735 Accurate Debug fps metrics + dropped-frame counter
- Sample every rate against a monotonic Stopwatch clock and read the camera
  frame count live, then EWMA-smooth, so delivered fps stops jittering
- Count genuinely lost frames at the capture (overwritten before the pipeline
  acquired them) and show the total over a rolling 60s window
- Tidy the Debug page header spacing
2026-10-27 19:01:40 -07:00
dan2wik d85fb961ff Add render-thread fps to Performance metrics and tidy labels
- Add a 'UI Render thread' row showing Avalonia's compositor frame rate,
  counted via a per-frame RequestCompositionUpdate hook in the view.
- Rename 'UI thread' to 'UI data thread' and the header to 'Performance
  metrics'; drop the (responsiveness) and (NNN% = fully saturated) notes.
- Relabel Face -> Mouth (inference rows, pipeline header, worker thread).
- Rename 'CPU hotspots (per thread)' to 'Thread CPU Usage' and
  '% of one core' to '% x 1 core'.
2026-10-27 19:01:39 -07:00
dan2wik 531f6f695f Fix camera freeze from Mat reuse in capture loop
The update loop reused one Mat that SetRawMat hands to the consumer, so
the next Read overwrote the buffer the consumer was still using and the
feed stalled. Allocate a fresh Mat per frame and dispose it on a miss.
2026-10-27 19:01:39 -07:00
dan2wik 8a76b5f6cd Revert "Fix Windows capture for cameras whose default DShow pin is Y800"
This reverts commit 5a2a7b10ce.
2026-10-27 19:01:39 -07:00
Rames The Generic 847f3af653 Add checkpoints for Linux Trainer 2026-10-27 19:01:38 -07:00
Rames The Generic 3daebd1001 Fix Stabilize Eyes not enabling 2026-10-27 19:01:38 -07:00
dan2wik 1edf79686c Fix Windows capture for cameras whose default DShow pin is Y800
OpenCV's DSHOW backend can't stream Y800 (raw greyscale) - which some
trackers expose as their default pin - and in fact can't open such
devices at all. Result was an endless "No data received" with the camera
never producing frames on Windows (it works on Linux via GStreamer/MJPEG).

- Add DirectShowModeSelector: enumerate a device's DirectShow modes and
  pick the best non-Y800 mode (prefer MJPG, then larger frame, then higher
  fps). Generic - no per-camera assumptions.
- OpenCvCapture: apply the selected mode as size -> fps -> FourCC (last),
  since the frame rate is what drives the backend onto the MJPG pin when a
  camera exposes the same resolution as both raw and compressed. Fall back
  to Media Foundation (MSMF) when DSHOW yields no frames; additive, so
  cameras already working on DSHOW are unaffected.
- Hand off a fresh Mat per frame in the capture loop. Reusing a single Mat
  raced the consumer (concurrent write / use-after-dispose) and crashed
  natively (0xC0000005) once frames actually started flowing.
- Log opened backend, negotiated format and a frame probe for diagnostics;
  sleep when idle to avoid busy-spinning a CPU core.
2026-10-27 19:01:37 -07:00
dan2wik 383650a9e9 Expression pipeline optimization + diagnostics
- ONNX: cap IntraOpNumThreads=2 to eliminate intra-op thread fan-out CPU.
  Eye-only CPU inference on a Ryzen 9 9900X: ~225% -> ~67% of one core
- Add always-on ThreadProfiler + Debug/Performance page: per-thread CPU,
  pipeline stage timings, throughput counters
- Settings: add "Split Eye Video Swap" (default unswapped) and opt-in
  "Show Debug menu"; move Advanced options to the bottom of the list
- Bundle in-progress expr-dev capture/pipeline work (V4L2 capture, video
  sources, OSC recv/send, parameter sender, home/calibration view models)
2026-10-27 19:01:37 -07:00
Rames The Generic 7d878b197c Add Alpha warning banner 2026-10-27 19:01:36 -07:00
Rames The Generic 1d9b3eeaf2 version number ig 2026-10-27 19:01:36 -07:00
Rames The Generic 11fbbcd39e Revert sequence times 2026-10-27 19:01:36 -07:00
Rames The Generic b4112fa3b4 Update calibration sequences and trainer 2026-10-27 19:01:35 -07:00