LiveKit Egress

Changelog

Added

• V2 egress API — unified source and output config (StartEgressRequest, MediaSource, Output, StorageConfig), per-participant audio channel routing (AudioRoute), and request-level storage overrides (#1155)
• Support for egress auto retry (#1138)
• Faster than realtime recording (non-live) (#1192)
• Cgroup-aware memory monitoring for admission control and OOM kill (#1118)
• Support for mulaw and alaw input codecs (#1105)
• Add handler outcome Prometheus metric for SLO-based kill rate alerting (#1230)
• Add livekit_load_ratio metric for composite load-based autoscaling (#1234)
• Better multi-publisher support (#1214)
• Read K8s CPU requests to automatically set GOMAXPROCS for cgroup-aware scheduling (#1204)
• Backup storage observability — including storage event IPC for tracking uploads across primary/backup stores (#1120, #1184)
• Enable one-shot sender report sync mode for room composite behind config (#1158)
• Export InitLogger with configurable service name (#1218)
• Allow specifying affinity timeout (#1104)
• Instrument all leaky queues (#1116)
• Instrument data loss on video leaky queues (#1109)
• Add a 1G memory buffer for accepting requests (#1088)
• Log rss periodically (#1151)
• Log pipeline time to playing (#1103)
• Log file upload stats (size, duration) (#1096)
• Chrome 146 (#1173)

Fixed

• Fix s3-compat multi-part uploads (#1228)
• Fix deadlock when AbortProcess/KillProcess call kill() under pm.mu lock (#1226)
• Use statistical cadence check to tolerate WebRTC NetEQ time-stretching (#1223)
• Fix MP3 duration metadata and CBR encoding (#1187)
• Fix room composite duration to include post-participant silence tail (#1169)
• Fix awaitMediaTracks race (#1165)
• Fix for x264 encoder errors causing egress failures at the end of the recording (#1095)
• Fix for data race inside monitor (#1091)
• Fixing one of the causes of pipeline frozen errors (#1129)
• Potential fix for rare issue causing tracks not to be recorded (#1130)
• Handle EOS when removing source bin (#1093)
• Drain appwriter (#1090)
• Drain audio tracks before removing their appsource (#1085)
• Heal track which enters into continuous flow flushing loop on pushing packets (#1142)
• One pacer per audio track (#1235)
• Set time provider only after pipeline reaches playing state (#1212)
• AbortProcess is not safe to be executed by multiple goroutines (#1208)
• Fixing unprotected state write from timer goroutine (#1206)
• Disable PTS adjustments on sender reports for track egresses (#1134)
• Disconnect from room on failing to await for some track (#1132)
• Retry chrome egress navigation on chrome cert verifier change (#1194)
• Return a 500 if handler fails to start (#1137)
• Make sure all data is read from LK server (#1111)
• Always check video dimensions (#1119)
• Fail web egress on HTTP 4xx/5xx page load errors (#1106)
• Avoid panic if we fail to parse a gst pipeline error (#1086)
• Address unsafe int casting (#1126)
• Get original room name from Start request (#1189)
• Suppress noisy colorimetry warnings (#1163)
• Don't log error on manifest upload failure if backup storage wasn't used (#1152)
• Don't count room composite SDK source against Pulse limits (#1114)
• Use a 10 min deadline for the RPC watchdog (#1207)
• Remove enable_room_composite_sdk_source and always enable sdk source when conditions allow it (#1122)
• Gstreamer logs based on configured level (#1161)
• Reintroduce sdk logs filtering (#1213)
• Using variant of OnDisconnected callback which passes a reason (#1221)
• AV sync content verification in integration tests (#1215)
• Fixing arm64 chrome installer (#1175)
• Use separate token with delete permission (#1172)
• Update module go.opentelemetry.io/otel to v1.41.0 [SECURITY] (#1201)
• Update module github.com/aws/aws-sdk-go-v2/service/s3 to v1.97.3 [SECURITY] (#1178)
• Update module github.com/go-jose/go-jose/v4 to v4.1.4 [SECURITY] (#1170)
• Update module google.golang.org/grpc to v1.79.3 [SECURITY] (#1153)
• Rename IOClient to SessionReporter (#1097)
• Move ProcessManager to interface and create a fake implementation (#1147)
• Enable staticcheck (#1094)

Changelog

Added

• Ability to request dot file dumping through gst callbacks (#1072)
• Invoke UpdateEgress on each stream retry attempt (#1055)
• Logging for file outputs exceeding set of file size thresholds (#1079)
• Ability for configuring max file size in bytes as a storage quota (#1080)
• Setting chrome.log debug file size limits & rotation (#1081)

Fixed

• Update eslint for js-yaml vuln fixes (#1074)
• Update protocol and x/crypto (#1082)
• Make stats meaningful for SRT and prevent log noise (#1075)
• Fix for racy addition of app sources (#1073)
• Ensure pipeline is at least in PAUSED state before pushing data to app sources by (#1076)

Changelog

Added

• Allow setting a default role ARN for S3 upload requests (#958)
• Agent testing (#960)
• Adding audio mixer QoS stats processing (#969)
• Add content checks to file integration tests (#988)
• Basic content checks for segment tests (#990)
• Handle small audio gaps by the signal time stretching/compression instead of having discontinuities (#974)
• Adding MP3 to the supported audio types for audio only recordings (#1007)
• Adding burst estimator (#1026)
• Adding gstreamer pipeline running/playhead time accessors (#1025)
• Adding latency overrides per request types (#1050)
• Add PSRPC observability to IOInfoClient (#1056)
• Add a watchdog to shutdown egress on PSRPC failures (#1061)

Fixed

• set status to failed when an egress is killed (#961)
• Log SDK app writer source stats on EOS (#963)
• Align test source output buffer duration with opus frame size (#966)
• disable rtcp for room composite (#971)
• Don't log every single audio discontinuity (#985)
• Switch from react-scripts to vite for egress-templates build (#989)
• Use specific template version with egress versions (#1000)
• Snap audio PTS back to the regular time grid (#1002)
• Speed up build time (#1004)
• Don't delete test results immediately - keep them for eventual after run analyisis (#1005)
• do not send eos on participant composite video tracks (#1008)
• Skip dummy video packets (#1013)
• Extend max allowed RTP timestamps diff (#1016)
• Use arrival time and one way delay estimation in PTS calculation. (#1015)
• Stair step discrete corrections of track start time (#1019)
• Update sdk to include quanitzed SRs adjustments (#1020)
• Go SDK for track synchronizer packet stats (#1021)
• Go SDK to set initial PTS to estimated PTS (#1022)
• Restore defaultRTPMaxAllowedTsDiff setting (#1023)
• Drop old samples when sample queue overflows. (#1024)
• Add controller nil checks to IPC handler (#1027)
• Update server-sdk to pull in the fix for initialize timing (#1029)
• RemoveTrack after finish. (#1030)
• Adding linters (#1033)
• Update server-sdk to pull in logs sampling fix (#1035)
• Update server-sdk to pull in the fix for ensuring no PTS regressions (#1039)
• Adding more logs around stopping pipeline (#1042)
• Update server-sdk to include the fix for sanitized SRs (#1043)
• Detect & handle missing PTS after vp9parse element (#1046)
• Adjusting UpdateEgress retry logic (#1048)
• VP9 - Drop a frame with an invalid PTS and wait for key frame (#1047
• Adding a workaround for baseparse gstreamer bugs which could produce buffer with no valid PTS (#1049)
• Fix filename templating in case of multi output (#1051)
• Log egressID in io client worker (#1054)
• Pay attention to context deadline on IO update (#1062)
• Close grpc ClientConn in the main process (#1064)
• Logging significant IPC callbacks (#1066)
• retry S3 upload on 409 or 429 (#1069)
• Disable scenecut when keyframe interval is set (#1070)

Changelog

• fix segment panic for very large durations (https://github.com/livekit/egress/pull/869)
• update logger (https://github.com/livekit/egress/pull/870) (https://github.com/livekit/egress/pull/874) (https://github.com/livekit/egress/pull/878) (https://github.com/livekit/egress/pull/894) (https://github.com/livekit/egress/pull/898) (https://github.com/livekit/egress/pull/918)
• fix panic on reading stream stats (https://github.com/livekit/egress/pull/881)
• fix for sdk room composite (https://github.com/livekit/egress/pull/883)
• do not log an empty string an error for glib errors/warnings (https://github.com/livekit/egress/pull/885)
• use core.Fuse in the source logic (https://github.com/livekit/egress/pull/888)
• do not fail image output if no dimension is set (https://github.com/livekit/egress/pull/890)
• support extra webhooks in egress requests (https://github.com/livekit/egress/pull/891)
• fix race between track subscriptions and audio bin creation (https://github.com/livekit/egress/pull/892)
• chrome logging + extended logging controls (https://github.com/livekit/egress/pull/897)
• add support for IMAGE_SUFFIX_NONE_OVERWRITE (https://github.com/livekit/egress/pull/895)
• update dependency @babel/runtime to v7.26.10 [SECURITY] (https://github.com/livekit/egress/pull/875)
• update all deps and go version (https://github.com/livekit/egress/pull/901)
• update template deps (https://github.com/livekit/egress/pull/902)
• update jitter buffer (https://github.com/livekit/egress/pull/903)
• fix audio gaps (https://github.com/livekit/egress/pull/905)
• use the livekit go-gst fork (https://github.com/livekit/egress/pull/904) (https://github.com/livekit/egress/pull/906) (https://github.com/livekit/egress/pull/933)
• update nth-check template dep (https://github.com/livekit/egress/pull/907)
• improved update management (https://github.com/livekit/egress/pull/909)
• SDK timing (https://github.com/livekit/egress/pull/910)
• update README yaml config regarding storage (https://github.com/livekit/egress/pull/916)
• move latencies to config (https://github.com/livekit/egress/pull/915)
• remove process handler on launch failure (https://github.com/livekit/egress/pull/917)
• nil check primary error (https://github.com/livekit/egress/pull/919)
• fix json ignore (https://github.com/livekit/egress/pull/920)
• fix audio mixer (https://github.com/livekit/egress/pull/924)
• adopt storage version of blob uploader (https://github.com/livekit/egress/pull/925)
• update storage, handler logger (https://github.com/livekit/egress/pull/930)
• feat(cli): update to urfave/cli/v3 (https://github.com/livekit/egress/pull/928)
• limit pulse clients (https://github.com/livekit/egress/pull/931)
• allow setting GST_DEBUG env at runtime (https://github.com/livekit/egress/pull/876)
• add GetAvailableMemory to monitor (https://github.com/livekit/egress/pull/932)
• fix azure urls (https://github.com/livekit/egress/pull/935)
• memory updates (https://github.com/livekit/egress/pull/936)
• delete the handler temporary context rehardless of the handler termination status (https://github.com/livekit/egress/pull/934)
• increase audio alignment threshold (https://github.com/livekit/egress/pull/937)
• fix race in audio track subscription (https://github.com/livekit/egress/pull/938)
• add duration to presigned urls (https://github.com/livekit/egress/pull/941)
• use gstreamer 1.24.12 (https://github.com/livekit/egress/pull/943)
• allow uploading to S3 using AssumeRole (https://github.com/livekit/egress/pull/940)
• update pion dependencies (https://github.com/livekit/egress/pull/946)
• fix for AssumeRole credential insertion (https://github.com/livekit/egress/pull/948)

Changelog

Added

• Support for backup storage (#792) (#793) (#795) (#825)
• Log per egress memory usage (#821)
• Support for participant egress screen share (#829)
• Use the SDK to handle audio only room composite requests with the default template (#817)

Fixed

• Skip AWS logging on successful upload (#762)
• Reduce duplicate test runs (#763)
• Fix s3logger memory usage (#764)
• Disable SDK Async-connect (#768)
• Faster startup (#769)
• Queue 15 min worth of segments (#770)
• Use config value for max upload queue size (#772)
• Disable-features Translate in headless Chrome (#775)
• Add a back off top laylist upload retries (#779)
• Remove audio with image-only egress (#782)
• Adopt AWS SDK v2 (#783)
• Force CBR for streams output (#785)
• Don't update bucket region with custom endpoints (#786)
• Set a region field when using a custom endpoint (#788)
• Ignore websocket close errors (#790)
• Avoid recording room composite before video is decoded (#806)
• Fix max concurrent web check (#810)
• Use Ubuntu 24.04 base image (#811)
• Move IO timeouts to config (#814)
• Upgrade to pion/webrtc v4 (#818)
• Kill highest memory egress when OOM (#826)
• Udpate livekit JS dependencies and call setRoom in default template (#835)

Changed

• Preset 0 (720p, 30fps) is now the default recording option (#640)

Fixed

• Zombie processes eventually causing chromium or X to fail (#673)
• Pipeline getting stuck with Participant composite when a track is unpublished (#635, #644)
• ImageOutputs broken when used with other outputs (#642)
• Fixes to jitter buffer and track synchronizer for track/track composite egress (#643, #660)
• Race on shutdown causing some final updates not to send (#666, #667, #674)

Added

• Audio-only cpu requirements (#597)
• is_disabled prometheus metric for terminating nodes (#622)
• Details field in EgressInfo (#627)

Fixed

• Race condition with deadlock (#608)
• Critical gstreamer error with Participant egress (#624, #626, #628)
• Keyframe intervals for segments and streams (#611, #618)
• multi output use case with image outputs and local image save path (#614)
• Force IPv4 with GCP client (#623)
• Marking aborted egress as failed (#620, #625)

Requires livekit-server v1.5.1+

Fixed

• Track/Track composite failures when video track begins muted (#524)
• Segment upload failures if queue gets backed up (#509)
• Sync issues (#498, #507)
• Stream reconnections (#503)

Fixed

• Pulse not closing properly eventually causing all room composite and web egress to fail (#486)

Fixed

• Improved RTMP retry policy (#444)
• Pipeline frozen errors (#459)
• Websocket egress marked as failed after being stopped (#450)
• Negative waitgroup counter bug with track/track composite egress (#463)

Added

• Supports decoding vp9 tracks with track/track composite egress (#465)

Changed

• Chrome sandboxing is now a config option, disabled by default (#481)

NOTE: as of v1.7.5, chrome sandboxing is enabled for increased security. As a result, the service is no longer run as root inside docker, and all Egress deployments (even local) require --cap-add=SYS_ADMIN in your docker run command. Without it, all web and room composite egress will fail with a chrome failed to start error.

Fixed

• Hang if EndRecording called before StartRecording (#392)
• Panic on concurrent websocket writes
• CPU overscheduling (leads to audio loss and other issues) (#396, #403, #406)
• Hang on gstreamer internal deadlock (#407)

Added

• Include logs from gstreamer (#398, #401, #411)
• Save space by deleting local segments once uploaded (#405)

Added

• AwaitStartSignal option for web egress (if true, it will not start until your website logs START_RECORDING)

Fixed

• New jitter buffer to help with track/track composite latency issues
• Updated track synchronizers to help with a/v sync issues
• Track websocket egress will send pings to keep websocket connection alive
• An instance will not schedule above 80% CPU (maxing out CPU causes issues)
• An instance will now accept a request if it's idle and requirement == total CPUs (for example, room_composite_cost = 4 and your pods have 4 CPU)

Changed

• Redacted stream keys in logs will show the starting and ending characters so they can be identified

Fixed

• State management bugs (#292, #333)
• Service stops replying for existing egress on shutdown (#297)
• Server panics (#301, #320, #348)
• Track sync/timestamping issues (#305, #313)
• Cost config not respected (#346)

Updated

• Templates use livekit-components (#323, #350)
• Better input validation (#322, #328)

Added

• Multiple outputs enabled (max one of each type) - for example, you can stream to multiple providers and store an mp4 all with one egress (currently only available by using twirp directly, sdk updates coming soon)
• Added timestamp suffix option with segmented file output, to match Agora's TS filename conventions

Fixed

• VP8 resolution change issue with track egress (Caps changes not supported by Mastroska error)
• Updated GStreamer to v1.22.0 for improved stability
• New internal communication system (PSRPC) with improved reliability - can be turned on by setting UsePSRPC to true in LiveKit Server config
• Various A/V sync and reliability improvements

Changed

• Google cloud storage config Credentials now takes a string instead of bytes to match the egress config

Changelog

Fixed

• Config refactoring https://github.com/livekit/egress/pull/173 & https://github.com/livekit/egress/pull/174
• Fix track websocket test https://github.com/livekit/egress/pull/176
• Better S3 credentials handling and validation https://github.com/livekit/egress/pull/177
• Fix issue where egress in state STARTING or ACTIVE would never change to COMPLETED
• Do not log failures due to misconfigured egress requests with "error" severity https://github.com/livekit/egress/pull/183
• Update logging library https://github.com/livekit/egress/pull/185
• Redact upload secrets before sending status events https://github.com/livekit/egress/pull/186
• Fix failure to unload pulseaudio plugin causing eventual pulseaudio failure.

Added

• Add process manager object to start and manage handler processes https://github.com/livekit/egress/pull/175
• Shutdown service if we hit a fatal error, such as pulseaudio failing https://github.com/livekit/egress/pull/181
• Log Chrome errors https://github.com/livekit/egress/pull/182

Changelog

Fixed

• support for running on linux systems with no cgroup support https://github.com/livekit/egress/pull/169
• CPU usage ratio reporting when a cgroup CPU quota is set https://github.com/livekit/egress/pull/169
• H264 subscription support with web composite exports on arm64 https://github.com/livekit/egress/pull/171

Changelog

Fixed

• support for older linux kernels with cgroup v1 #155
• default HLS segment and playlist filename #166
• S3 upload metadata #156
• fix for disabling storing manifest #159
• interaction with Digital Ocean S3 Implementation #162

Added

• support for Aliyun OSS storage backend #152
• support for web egress #155

Fixed

• Choppy audio when streaming to Youtube (#107)
• Added support for CJK unicode characters in templates (#98)
• Allow query parameters in custom template base (#99)

Changed

• Upgraded Gstreamer from 1.18.5 to 1.20.3 (#102)
• Upgraded default layouts (#100)

Fixed

• Rare freezing and/or segmentation faults, primarily with track composite (#92) (#93) (#94)

Changed

• Egress stopped due to session limit now returns LIMIT_REACHED status instead of FAILED (#90)
• Track egress for vp8 tracks will output a .webm file instead of .ivf (#96)

Added

• Support for session timeouts (#76)
• Redis sentinel support (#71)

Fixed

• File upload not running with track egress (#70)
• Bug where StopEgress RPCs return "no response" for track and track composite egress (#71)
• Failure when track is not immediately available with track and track composite egress (#72)
• Test suite reliability, along with testing cloud uploads (#71)