mirror of
https://github.com/rustdesk/hbb_common.git
synced 2026-08-28 13:01:12 +00:00
webrtc: own every peer connection's I/O on a process-lifetime runtime
A pc's UDP sockets register with the reactor, and its ICE/DTLS/SCTP pumps spawn on the runtime, that is current while `new` builds it. For a rustdesk controller that is io_loop's own `#[tokio::main(flavor = "current_thread")]` runtime, dropped the moment io_loop returns — so the pc outlived the only runtime able to drive its I/O, and the session-end close, merely spawned there, was never polled once. No DTLS close_notify left, and the peer waited out ICE decay (~25-30s in its log) where TCP delivers a FIN at once. Driving the close from somewhere else does not help: the sockets and pumps are already gone, so close() returns having swallowed every transport error without reaching the wire. Home the pc where it can outlive its caller instead. `new` runs its body on WEBRTC_RT, a lazy process-lifetime runtime, so every socket and background task belongs to it; `close_detached_with` spawns each close there as its own task — never cancelled, since a close cancelled after RTCPeerConnection::close latches `is_closed` is unretryable and strands the pc in SESSIONS, and never serialized, since close() has no deadline and one stalled pc would otherwise block every later session's teardown. Data-plane futures are still polled from caller runtimes; only the owning driver has to stay alive. A cancelled `new` needs the same care: rt.spawn keeps running when its JoinHandle is dropped, so the setup task would finish, cache the session, and hand its result to nobody — and an unanswered offerer never reaches a terminal ICE state, so it sat in SESSIONS forever. NewStreamHandoff closes an abandoned freshly-built stream, and hands a cache hit back disarmed: that one is shared with a live caller whose connection must survive this one's cancellation. Tests cover the production shapes: a session-end close from a thread with no runtime, an EOF-then-Drop double close, a close queued as the creating runtime is destroyed, and the cancelled-new handoff both ways. Two suite leaks surfaced on the way — streams dropped without close, polluting SESSIONS for every later test — and are fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HV43uh1ztv6Wm5qi3Y1ne
This commit is contained in:
+3
-1
@@ -95,7 +95,9 @@ impl Stream {
|
|||||||
/// abandon, and an awaited close that loses that race is unretryable — `close()` latches
|
/// abandon, and an awaited close that loses that race is unretryable — `close()` latches
|
||||||
/// `is_closed` before its first await, so every later attempt early-returns while the state
|
/// `is_closed` before its first await, so every later attempt early-returns while the state
|
||||||
/// handler that would evict the session never runs. With no await point here there is
|
/// handler that would evict the session never runs. With no await point here there is
|
||||||
/// nothing to cancel; the teardown finishes on the runtime.
|
/// nothing to cancel; the teardown runs to completion on the WebRTC I/O runtime
|
||||||
|
/// (see `webrtc::WEBRTC_RT`) that owns the pc's sockets and pump tasks, so it reaches the
|
||||||
|
/// wire even after a caller's own runtime dies on return.
|
||||||
#[inline]
|
#[inline]
|
||||||
pub fn close_webrtc(&self) {
|
pub fn close_webrtc(&self) {
|
||||||
match self {
|
match self {
|
||||||
|
|||||||
+405
-44
@@ -19,7 +19,7 @@
|
|||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use std::io::{Error, ErrorKind};
|
use std::io::{Error, ErrorKind};
|
||||||
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
|
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
|
||||||
use std::sync::{Arc, Mutex as StdMutex};
|
use std::sync::{Arc, Mutex as StdMutex};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
@@ -79,6 +79,28 @@ pub struct WebRTCStream {
|
|||||||
// matched to the signed peer id, via `set_key`). DTLS always encrypts; this flag mirrors TCP's
|
// matched to the signed peer id, via `set_key`). DTLS always encrypts; this flag mirrors TCP's
|
||||||
// "secured after key exchange" so key-less / unbound WebRTC is not shown as peer-authenticated.
|
// "secured after key exchange" so key-less / unbound WebRTC is not shown as peer-authenticated.
|
||||||
peer_verified: Arc<AtomicBool>,
|
peer_verified: Arc<AtomicBool>,
|
||||||
|
// Whether anyone still wants this pc — see `HandoffState`. Shared by every clone, the
|
||||||
|
// `SESSIONS` entry included, because that question is about the pc, not about one handle.
|
||||||
|
handoff: Arc<HandoffState>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether a pc `new_inner` built is still wanted, shared by every `NewStreamHandoff` handed out
|
||||||
|
/// for it. `new_inner` caches the pc before its `new()` returns, so the builder's handoff and any
|
||||||
|
/// number of cache hits are in flight at once; a pc is abandoned only when the last of them is
|
||||||
|
/// dropped with none adopted, and closing on any single Drop would kill a live caller's session.
|
||||||
|
#[derive(Default)]
|
||||||
|
struct HandoffState {
|
||||||
|
// Handoffs handed out and neither adopted nor dropped yet. Incremented under the `SESSIONS`
|
||||||
|
// lock so a hit that is still arriving cannot be overtaken by the abandon-close below.
|
||||||
|
pending: AtomicUsize,
|
||||||
|
// Latched by the first `into_inner`: some caller owns the stream, and since `WebRTCStream`
|
||||||
|
// has no `Drop` there is nothing that could un-own it. Abandoning any later hit is then a
|
||||||
|
// no-op, which is what keeps a cached pc alive across a cancelled caller.
|
||||||
|
adopted: AtomicBool,
|
||||||
|
// Set under the `SESSIONS` lock once the abandon-close commits, so a hit that locks the map
|
||||||
|
// afterwards (eviction only happens later, from the state handler) rejects the entry instead
|
||||||
|
// of adopting a pc that is already going away.
|
||||||
|
closing: AtomicBool,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Default)]
|
#[derive(Default)]
|
||||||
@@ -126,6 +148,69 @@ static DEFAULT_ICE_SERVERS: [&str; 3] = [
|
|||||||
|
|
||||||
lazy_static::lazy_static! {
|
lazy_static::lazy_static! {
|
||||||
static ref SESSIONS: Arc::<Mutex<HashMap<String, WebRTCStream>>> = Default::default();
|
static ref SESSIONS: Arc::<Mutex<HashMap<String, WebRTCStream>>> = Default::default();
|
||||||
|
|
||||||
|
// The process-lifetime runtime that owns ALL WebRTC I/O. A pc's UDP sockets register with
|
||||||
|
// the reactor — and its ICE/DTLS/SCTP pump tasks spawn on the runtime — that is current
|
||||||
|
// while `new_inner` and the handlers it installs execute; if that were a session's
|
||||||
|
// `#[tokio::main]` runtime (dropped the moment io_loop returns), the sockets and pumps
|
||||||
|
// would die with it and even a close driven elsewhere would "succeed" without ever putting
|
||||||
|
// close_notify on the wire — the peer then waits out ICE decay. So `new()` runs its body
|
||||||
|
// here, and detached closes run here too: as independent tasks (a deadline-free `close()`
|
||||||
|
// must not head-of-line block other sessions' teardown) that are never cancelled (a close
|
||||||
|
// cancelled after `RTCPeerConnection::close` latches `is_closed` is unretryable, stranding
|
||||||
|
// the pc in SESSIONS). Data-plane futures (send/recv/close) may be polled from any caller
|
||||||
|
// runtime — cross-runtime polling is fine; only the OWNING driver must stay alive. Two
|
||||||
|
// workers so one busy session cannot starve the rest. This is the one deliberate exception
|
||||||
|
// to AGENTS.md's no-runtime-in-libraries rule: a single lazily-built runtime for I/O whose
|
||||||
|
// lifetime exceeds any caller's.
|
||||||
|
static ref WEBRTC_RT: Option<tokio::runtime::Runtime> = tokio::runtime::Builder::new_multi_thread()
|
||||||
|
.worker_threads(2)
|
||||||
|
.thread_name("webrtc-io")
|
||||||
|
.enable_all()
|
||||||
|
.build()
|
||||||
|
.map_err(|err| log::error!("failed to build the WebRTC I/O runtime: {}", err))
|
||||||
|
.ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hands a stream from `new_inner` (running detached on `WEBRTC_RT`) back to its `new()`
|
||||||
|
/// caller. If that caller was cancelled while awaiting, the runtime drops the task's output —
|
||||||
|
/// this guard — and an unanswered offerer never reaches a terminal ICE state on its own, so
|
||||||
|
/// without the Drop-close its pc, sockets and pump tasks would sit in SESSIONS forever. Claims
|
||||||
|
/// are counted in the shared `HandoffState`, so only the last one dropped with the stream still
|
||||||
|
/// unadopted closes it; a hit another caller is already using is left alone.
|
||||||
|
struct NewStreamHandoff {
|
||||||
|
stream: Option<WebRTCStream>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl NewStreamHandoff {
|
||||||
|
/// Take a claim on `stream`. Call under the `SESSIONS` lock: a claim registered after the map
|
||||||
|
/// is unlocked can land behind the abandon-close of the claim that was last outstanding.
|
||||||
|
fn claim(stream: WebRTCStream) -> Self {
|
||||||
|
stream.handoff.pending.fetch_add(1, Ordering::AcqRel);
|
||||||
|
Self {
|
||||||
|
stream: Some(stream),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn into_inner(mut self) -> Option<WebRTCStream> {
|
||||||
|
let stream = self.stream.take()?;
|
||||||
|
// Before releasing the claim, or a concurrent Drop can see the count reach zero without
|
||||||
|
// seeing that this caller took ownership.
|
||||||
|
stream.handoff.adopted.store(true, Ordering::Release);
|
||||||
|
stream.handoff.pending.fetch_sub(1, Ordering::AcqRel);
|
||||||
|
Some(stream)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for NewStreamHandoff {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
if let Some(stream) = self.stream.take() {
|
||||||
|
let last = stream.handoff.pending.fetch_sub(1, Ordering::AcqRel) == 1;
|
||||||
|
if last && !stream.handoff.adopted.load(Ordering::Acquire) {
|
||||||
|
stream.close_if_abandoned();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Clone for WebRTCStream {
|
impl Clone for WebRTCStream {
|
||||||
@@ -142,6 +227,7 @@ impl Clone for WebRTCStream {
|
|||||||
send_gate: self.send_gate.clone(),
|
send_gate: self.send_gate.clone(),
|
||||||
recv_state: self.recv_state.clone(),
|
recv_state: self.recv_state.clone(),
|
||||||
peer_verified: self.peer_verified.clone(),
|
peer_verified: self.peer_verified.clone(),
|
||||||
|
handoff: self.handoff.clone(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -239,6 +325,7 @@ impl WebRTCStream {
|
|||||||
/// evicts late enough that dead entries linger. Both the lookup and the insert must use it.
|
/// evicts late enough that dead entries linger. Both the lookup and the insert must use it.
|
||||||
fn is_reusable_for(&self, force_relay: bool) -> bool {
|
fn is_reusable_for(&self, force_relay: bool) -> bool {
|
||||||
self.relay_only == force_relay
|
self.relay_only == force_relay
|
||||||
|
&& !self.handoff.closing.load(Ordering::Acquire)
|
||||||
&& !matches!(
|
&& !matches!(
|
||||||
*self.state_notify.borrow(),
|
*self.state_notify.borrow(),
|
||||||
WebRTCConnectionState::Closed(_)
|
WebRTCConnectionState::Closed(_)
|
||||||
@@ -401,11 +488,31 @@ impl WebRTCStream {
|
|||||||
ice_servers
|
ice_servers
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Built and driven on `WEBRTC_RT`: every socket and background task the pc creates must
|
||||||
|
/// belong to a runtime that outlives the session (see `WEBRTC_RT`). A caller that abandons
|
||||||
|
/// this future mid-await leaves the setup task to finish there; the abandoned result then
|
||||||
|
/// closes a freshly built pc (and leaves a shared cache hit alone) — see `NewStreamHandoff`.
|
||||||
pub async fn new(
|
pub async fn new(
|
||||||
remote_endpoint: &str,
|
remote_endpoint: &str,
|
||||||
force_relay: bool,
|
force_relay: bool,
|
||||||
ms_timeout: u64,
|
ms_timeout: u64,
|
||||||
) -> ResultType<Self> {
|
) -> ResultType<Self> {
|
||||||
|
let Some(rt) = WEBRTC_RT.as_ref() else {
|
||||||
|
return Err(anyhow::anyhow!("WebRTC I/O runtime unavailable"));
|
||||||
|
};
|
||||||
|
let remote_endpoint = remote_endpoint.to_owned();
|
||||||
|
rt.spawn(Self::new_inner(remote_endpoint, force_relay, ms_timeout))
|
||||||
|
.await
|
||||||
|
.map_err(|err| anyhow::anyhow!("WebRTC setup task failed: {}", err))??
|
||||||
|
.into_inner()
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("WebRTC setup handoff was empty"))
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn new_inner(
|
||||||
|
remote_endpoint: String,
|
||||||
|
force_relay: bool,
|
||||||
|
ms_timeout: u64,
|
||||||
|
) -> ResultType<NewStreamHandoff> {
|
||||||
// The endpoint contains a Base64-encoded SDP with host addresses and live ICE
|
// The endpoint contains a Base64-encoded SDP with host addresses and live ICE
|
||||||
// credentials. Log only its size so debug logs cannot disclose that information.
|
// credentials. Log only its size so debug logs cannot disclose that information.
|
||||||
log::debug!(
|
log::debug!(
|
||||||
@@ -415,28 +522,34 @@ impl WebRTCStream {
|
|||||||
let remote_offer = if remote_endpoint.is_empty() {
|
let remote_offer = if remote_endpoint.is_empty() {
|
||||||
"".into()
|
"".into()
|
||||||
} else {
|
} else {
|
||||||
Self::get_remote_offer(remote_endpoint)?
|
Self::get_remote_offer(&remote_endpoint)?
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut key = Self::get_key_for_sdp_json(&remote_offer)?;
|
let mut key = Self::get_key_for_sdp_json(&remote_offer)?;
|
||||||
let start_local_offer = remote_offer.is_empty();
|
let start_local_offer = remote_offer.is_empty();
|
||||||
if !key.is_empty() {
|
if !key.is_empty() {
|
||||||
|
// Claim inside the lock: `close_if_abandoned` commits under it too, so this hit
|
||||||
|
// either registers in time to call that close off, or finds the entry already rejected.
|
||||||
let cached = {
|
let cached = {
|
||||||
let sessions_lock = SESSIONS.lock().await;
|
let sessions_lock = SESSIONS.lock().await;
|
||||||
sessions_lock
|
match sessions_lock.get(&Self::cache_key(&key, start_local_offer)) {
|
||||||
.get(&Self::cache_key(&key, start_local_offer))
|
Some(session) if session.is_reusable_for(force_relay) => {
|
||||||
.cloned()
|
Some(NewStreamHandoff::claim(session.clone()))
|
||||||
};
|
}
|
||||||
if let Some(cached_stream) = cached {
|
Some(session) => {
|
||||||
if cached_stream.is_reusable_for(force_relay) {
|
log::debug!(
|
||||||
log::debug!("Start webrtc with cached peer");
|
"Ignoring cached webrtc peer (relay_only {}, wanted {})",
|
||||||
return Ok(cached_stream);
|
session.relay_only,
|
||||||
|
force_relay
|
||||||
|
);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
None => None,
|
||||||
}
|
}
|
||||||
log::debug!(
|
};
|
||||||
"Ignoring cached webrtc peer (relay_only {}, wanted {})",
|
if let Some(handoff) = cached {
|
||||||
cached_stream.relay_only,
|
log::debug!("Start webrtc with cached peer");
|
||||||
force_relay
|
return Ok(handoff);
|
||||||
);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Create a SettingEngine and enable Detach
|
// Create a SettingEngine and enable Detach
|
||||||
@@ -682,31 +795,38 @@ impl WebRTCStream {
|
|||||||
send_gate: Arc::new(Semaphore::new(1)),
|
send_gate: Arc::new(Semaphore::new(1)),
|
||||||
recv_state: Arc::new(Mutex::new(RecvState::default())),
|
recv_state: Arc::new(Mutex::new(RecvState::default())),
|
||||||
peer_verified: Arc::new(AtomicBool::new(false)),
|
peer_verified: Arc::new(AtomicBool::new(false)),
|
||||||
|
handoff: Default::default(),
|
||||||
};
|
};
|
||||||
// Insert into the session cache, but never `await pc.close()` while holding this lock:
|
// Insert into the session cache, but never `await pc.close()` while holding this lock:
|
||||||
// `close()` fires the peer-connection-state handler inline, which itself locks SESSIONS,
|
// `close()` fires the peer-connection-state handler inline, which itself locks SESSIONS,
|
||||||
// self-deadlocking the whole process. Resolve any duplicate off-lock.
|
// self-deadlocking the whole process. Resolve any duplicate off-lock.
|
||||||
let cache_key = Self::cache_key(&key, start_local_offer);
|
let cache_key = Self::cache_key(&key, start_local_offer);
|
||||||
let duplicate = {
|
// Claim inside the lock, the insert included: with the pc cached but unclaimed, a hit
|
||||||
|
// could take the only claim on it and release it again, closing it under this caller.
|
||||||
|
let mut duplicate = None;
|
||||||
|
let handoff = {
|
||||||
let mut final_lock = SESSIONS.lock().await;
|
let mut final_lock = SESSIONS.lock().await;
|
||||||
// Same admissibility test as the lookup above, or that lookup is dead code: an entry
|
// Same admissibility test as the lookup above, or that lookup is dead code: an entry
|
||||||
// rejected there is still in the map when we get here, so returning it unconditionally
|
// rejected there is still in the map when we get here, so returning it unconditionally
|
||||||
// would discard the pc we just built precisely because the cached one was unusable.
|
// would discard the pc we just built precisely because the cached one was unusable.
|
||||||
match final_lock.get(&cache_key) {
|
match final_lock.get(&cache_key) {
|
||||||
Some(session) if session.is_reusable_for(force_relay) => Some(session.clone()),
|
Some(session) if session.is_reusable_for(force_relay) => {
|
||||||
|
duplicate = Some(webrtc_stream.clone());
|
||||||
|
NewStreamHandoff::claim(session.clone())
|
||||||
|
}
|
||||||
_ => {
|
_ => {
|
||||||
final_lock.insert(cache_key, webrtc_stream.clone());
|
final_lock.insert(cache_key, webrtc_stream.clone());
|
||||||
None
|
NewStreamHandoff::claim(webrtc_stream)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
if let Some(session) = duplicate {
|
if let Some(loser) = duplicate {
|
||||||
// A concurrent `new()` already cached an equivalent stream; discard this pc's
|
// A concurrent `new()` already cached an equivalent stream; discard this pc's
|
||||||
// resources (off-lock) and return the cached one.
|
// resources (on the closer thread — awaiting here would strand the pc if this
|
||||||
webrtc_stream.close().await;
|
// `new()` were cancelled mid-close) and return the cached one.
|
||||||
return Ok(session);
|
loser.close_detached();
|
||||||
}
|
}
|
||||||
Ok(webrtc_stream)
|
Ok(handoff)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One-shot endpoint: waits for ICE gathering so the SDP already carries the candidates.
|
/// One-shot endpoint: waits for ICE gathering so the SDP already carries the candidates.
|
||||||
@@ -852,27 +972,23 @@ impl WebRTCStream {
|
|||||||
self.pc.close().await.ok();
|
self.pc.close().await.ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Tear the pc down on the runtime, keeping `keep` alive until the teardown finishes.
|
/// Tear the pc down as an independent task on `WEBRTC_RT` — the runtime that owns the
|
||||||
///
|
/// pc's sockets and pump tasks, so the close can actually reach the wire no matter which
|
||||||
/// Callers poll this path inside a `select!`, and `close()` latches `is_closed` before its
|
/// caller runtimes have died — keeping `keep` alive until the teardown finishes. Callable
|
||||||
/// first await, so a cancelled close strands the pc: unretryable, never evicted from
|
/// from any context, `Drop` on a runtime-less thread included, and never cancelled, so
|
||||||
/// `SESSIONS`, never notifying `Closed`. The send path passes its permit as `keep`.
|
/// `close()`'s `is_closed` latch is only ever set by a close that finishes. The send path
|
||||||
|
/// passes its permit as `keep`.
|
||||||
pub fn close_detached_with<T: Send + 'static>(&self, keep: T) {
|
pub fn close_detached_with<T: Send + 'static>(&self, keep: T) {
|
||||||
let pc = self.pc.clone();
|
// If the runtime never built, no pc exists either (`new` fails first); nothing to close.
|
||||||
// Take the runtime handle explicitly rather than calling `tokio::spawn`: this also runs
|
let Some(rt) = WEBRTC_RT.as_ref() else {
|
||||||
// from `Drop`, which can execute on a thread with no runtime, where a bare spawn panics —
|
|
||||||
// and a panic in a destructor aborts the process. Without a handle the pc cannot be closed
|
|
||||||
// here at all; it is released at process exit. (Catching a panic from `Handle::spawn`
|
|
||||||
// during runtime shutdown is not an option: release builds set `panic = "abort"`.)
|
|
||||||
let Ok(handle) = tokio::runtime::Handle::try_current() else {
|
|
||||||
log::debug!("no tokio runtime available to close the WebRTC peer connection");
|
|
||||||
return;
|
return;
|
||||||
};
|
};
|
||||||
handle.spawn(async move {
|
let pc = self.pc.clone();
|
||||||
let _keep = keep;
|
rt.spawn(async move {
|
||||||
if let Err(err) = pc.close().await {
|
if let Err(err) = pc.close().await {
|
||||||
log::debug!("WebRTC background close failed: {}", err);
|
log::debug!("WebRTC close failed: {}", err);
|
||||||
}
|
}
|
||||||
|
drop(keep);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -881,6 +997,32 @@ impl WebRTCStream {
|
|||||||
self.close_detached_with(());
|
self.close_detached_with(());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Close a pc no caller ever took. Spawned on `WEBRTC_RT` like `close_detached`, but the
|
||||||
|
/// decision is remade under the `SESSIONS` lock that a cache hit claims under: either the hit
|
||||||
|
/// registers first and this call stands down, or `closing` commits first and
|
||||||
|
/// `is_reusable_for` rejects the entry, which the state handler evicts only later.
|
||||||
|
fn close_if_abandoned(&self) {
|
||||||
|
let Some(rt) = WEBRTC_RT.as_ref() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let stream = self.clone();
|
||||||
|
rt.spawn(async move {
|
||||||
|
{
|
||||||
|
let _sessions_lock = SESSIONS.lock().await;
|
||||||
|
if stream.handoff.pending.load(Ordering::Acquire) != 0
|
||||||
|
|| stream.handoff.adopted.load(Ordering::Acquire)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
stream.handoff.closing.store(true, Ordering::Release);
|
||||||
|
}
|
||||||
|
// Off-lock: `close()` fires the state handler inline and that handler locks SESSIONS.
|
||||||
|
if let Err(err) = stream.pc.close().await {
|
||||||
|
log::debug!("WebRTC close of an abandoned peer failed: {}", err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
#[inline]
|
#[inline]
|
||||||
pub fn set_raw(&mut self) {
|
pub fn set_raw(&mut self) {
|
||||||
// not-supported
|
// not-supported
|
||||||
@@ -1188,7 +1330,7 @@ pub fn is_webrtc_endpoint(endpoint: &str) -> bool {
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use crate::webrtc::WebRTCStream;
|
use crate::webrtc::WebRTCStream;
|
||||||
use crate::webrtc::{DEFAULT_ICE_SERVERS, FRAG_MORE, SESSIONS};
|
use crate::webrtc::{NewStreamHandoff, DEFAULT_ICE_SERVERS, FRAG_MORE, SESSIONS, WEBRTC_RT};
|
||||||
use bytes::{BufMut, Bytes, BytesMut};
|
use bytes::{BufMut, Bytes, BytesMut};
|
||||||
use std::{sync::Arc, time::Duration};
|
use std::{sync::Arc, time::Duration};
|
||||||
use tokio::sync::Barrier;
|
use tokio::sync::Barrier;
|
||||||
@@ -1489,10 +1631,12 @@ IHR5cCBzcmZseCByYWRkciAwLjAuMC4wIHJwb3J0IDY0MDA4XHJcbmE9ZW5kLW9mLWNhbmRpZGF0ZXNc
|
|||||||
"invalid webrtc endpoint should error"
|
"invalid webrtc endpoint should error"
|
||||||
);
|
);
|
||||||
|
|
||||||
assert!(
|
let stream = WebRTCStream::new("", false, 10000)
|
||||||
WebRTCStream::new("", false, 10000).await.is_ok(),
|
.await
|
||||||
"local webrtc endpoint should ok"
|
.expect("local webrtc endpoint should ok");
|
||||||
);
|
// A raw WebRTCStream has no Drop: close it, or its session stays cached in SESSIONS
|
||||||
|
// and pollutes cross-test assertions about the cache.
|
||||||
|
stream.close().await;
|
||||||
|
|
||||||
endpoint = "webrtc://eyJ0eXBlIjoiYW5zd2VyIiwic2RwIjoidj0wXHJcbm89LSA0MTA1NDk3NTY2NDgyMTQzODEwIDYwMzk1NzQw\
|
endpoint = "webrtc://eyJ0eXBlIjoiYW5zd2VyIiwic2RwIjoidj0wXHJcbm89LSA0MTA1NDk3NTY2NDgyMTQzODEwIDYwMzk1NzQw\
|
||||||
MCBJTiBJUDQgMC4wLjAuMFxyXG5zPS1cclxudD0wIDBcclxuYT1maW5nZXJwcmludDpzaGEtMjU2IDYxOjYwOjc0OjQwOjI4OkNFOjBCOjBDOjc1OjRCOj\
|
MCBJTiBJUDQgMC4wLjAuMFxyXG5zPS1cclxudD0wIDBcclxuYT1maW5nZXJwcmludDpzaGEtMjU2IDYxOjYwOjc0OjQwOjI4OkNFOjBCOjBDOjc1OjRCOj\
|
||||||
@@ -1515,6 +1659,7 @@ IHR5cCBzcmZseCByYWRkciAwLjAuMC4wIHJwb3J0IDY0MDA4XHJcbmE9ZW5kLW9mLWNhbmRpZGF0ZXNc
|
|||||||
let mut stream = WebRTCStream::new("", false, 100).await.unwrap();
|
let mut stream = WebRTCStream::new("", false, 100).await.unwrap();
|
||||||
let err = stream.wait_connected(10).await.unwrap_err();
|
let err = stream.wait_connected(10).await.unwrap_err();
|
||||||
assert!(err.to_string().contains("timeout"));
|
assert!(err.to_string().contains("timeout"));
|
||||||
|
stream.close().await; // see test_webrtc_new_stream: no Drop on a raw WebRTCStream
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn connect_loopback() -> (WebRTCStream, WebRTCStream) {
|
async fn connect_loopback() -> (WebRTCStream, WebRTCStream) {
|
||||||
@@ -1816,6 +1961,222 @@ IHR5cCBzcmZseCByYWRkciAwLjAuMC4wIHJwb3J0IDY0MDA4XHJcbmE9ZW5kLW9mLWNhbmRpZGF0ZXNc
|
|||||||
.expect("webrtc loopback did not complete in time");
|
.expect("webrtc loopback did not complete in time");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A client session runs on its own `#[tokio::main(flavor = "current_thread")]` runtime that
|
||||||
|
// is dropped the moment io_loop returns, so a close spawned or awaited there can be killed
|
||||||
|
// before (or worse, after) `close()` latches `is_closed`. Pin the production sequence over
|
||||||
|
// `Stream::WebRTC`: `close_webrtc()` from a thread with no runtime at all, then Drop; the
|
||||||
|
// peer must still see EOF — the closer thread, not any caller runtime, delivers it.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn test_session_end_close_reaches_the_peer() {
|
||||||
|
let (offerer, mut answerer) = connect_loopback().await;
|
||||||
|
let stream = crate::Stream::WebRTC(offerer);
|
||||||
|
tokio::task::spawn_blocking(move || {
|
||||||
|
stream.close_webrtc();
|
||||||
|
drop(stream);
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.expect("session thread");
|
||||||
|
|
||||||
|
match timeout(Duration::from_secs(5), answerer.next()).await {
|
||||||
|
Ok(None) | Ok(Some(Err(_))) => {}
|
||||||
|
Ok(Some(Ok(b))) => panic!("expected EOF after session close, got {} bytes", b.len()),
|
||||||
|
Err(_) => panic!("peer never observed the session-end close"),
|
||||||
|
}
|
||||||
|
answerer.close().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The shared count, not any one handoff, decides: with a second claim outstanding on a pc
|
||||||
|
// no one has adopted, dropping the first must leave it open for that caller, and dropping
|
||||||
|
// the last must close it — an unanswered offerer never reaches a terminal ICE state itself.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn test_last_abandoned_claim_closes_an_unadopted_pc() {
|
||||||
|
let builder = WEBRTC_RT
|
||||||
|
.as_ref()
|
||||||
|
.expect("WebRTC I/O runtime")
|
||||||
|
.spawn(WebRTCStream::new_inner(String::new(), false, 20000))
|
||||||
|
.await
|
||||||
|
.expect("setup task")
|
||||||
|
.expect("offerer builds");
|
||||||
|
let stream = builder
|
||||||
|
.stream
|
||||||
|
.as_ref()
|
||||||
|
.expect("a fresh handoff carries its stream")
|
||||||
|
.clone();
|
||||||
|
let key = format!("offer:{}", stream.session_key());
|
||||||
|
let hit = NewStreamHandoff::claim(stream);
|
||||||
|
|
||||||
|
drop(builder);
|
||||||
|
tokio::time::sleep(Duration::from_millis(300)).await;
|
||||||
|
assert!(
|
||||||
|
SESSIONS.lock().await.contains_key(&key),
|
||||||
|
"a claim was still outstanding, so the pc had to stay open"
|
||||||
|
);
|
||||||
|
|
||||||
|
drop(hit);
|
||||||
|
for _ in 0..200 {
|
||||||
|
if !SESSIONS.lock().await.contains_key(&key) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||||
|
}
|
||||||
|
panic!("the last abandoned claim never closed the unadopted pc");
|
||||||
|
}
|
||||||
|
|
||||||
|
// `new_inner` caches the pc BEFORE its own `new()` returns, so a concurrent caller can hit
|
||||||
|
// that cache and adopt the stream while the builder's handoff is still outstanding. Closing
|
||||||
|
// on the builder's Drop then tears down a connection another caller is already using, so
|
||||||
|
// abandonment has to be a property of the pc, not of one handoff.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn test_abandoned_builder_handoff_spares_an_adopted_pc() {
|
||||||
|
// Exactly what `new()` does, minus the adoption: the setup task runs on WEBRTC_RT and
|
||||||
|
// hands back a fresh, still-unadopted handoff.
|
||||||
|
let builder = WEBRTC_RT
|
||||||
|
.as_ref()
|
||||||
|
.expect("WebRTC I/O runtime")
|
||||||
|
.spawn(WebRTCStream::new_inner(String::new(), false, 20000))
|
||||||
|
.await
|
||||||
|
.expect("setup task")
|
||||||
|
.expect("offerer builds");
|
||||||
|
let stream = builder
|
||||||
|
.stream
|
||||||
|
.as_ref()
|
||||||
|
.expect("a fresh handoff carries its stream")
|
||||||
|
.clone();
|
||||||
|
let key = format!("offer:{}", stream.session_key());
|
||||||
|
assert!(
|
||||||
|
SESSIONS.lock().await.contains_key(&key),
|
||||||
|
"new_inner must cache the pc it built"
|
||||||
|
);
|
||||||
|
|
||||||
|
// The concurrent `new()`: a cache hit that takes the stream and goes on using it.
|
||||||
|
let adopted = NewStreamHandoff::claim(stream)
|
||||||
|
.into_inner()
|
||||||
|
.expect("a cache hit hands the stream back");
|
||||||
|
// ...and now the builder's `new()` is cancelled, so the runtime drops its handoff.
|
||||||
|
drop(builder);
|
||||||
|
|
||||||
|
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||||
|
assert!(
|
||||||
|
SESSIONS.lock().await.contains_key(&key),
|
||||||
|
"abandoning the builder's handoff closed a pc another caller had already adopted"
|
||||||
|
);
|
||||||
|
adopted.close_detached();
|
||||||
|
}
|
||||||
|
|
||||||
|
// End-to-end: cancel `new()` itself after its first poll (the spawn is already in flight)
|
||||||
|
// and every session it transiently created must be closed and evicted again. Keys are
|
||||||
|
// compared as a set difference so concurrent tests' own sessions do not interfere; theirs
|
||||||
|
// clean up within the wait too.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn test_cancelled_new_does_not_leak_the_pc() {
|
||||||
|
use std::collections::HashSet;
|
||||||
|
let before: HashSet<String> = SESSIONS.lock().await.keys().cloned().collect();
|
||||||
|
// Zero timeout: polls the future exactly once (spawning new_inner), then cancels it.
|
||||||
|
let _ = timeout(Duration::ZERO, WebRTCStream::new("", false, 20000)).await;
|
||||||
|
// Let the detached setup task finish (and insert its session) before demanding the
|
||||||
|
// difference be empty, or an early check passes vacuously while the leak forms later.
|
||||||
|
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||||
|
// 30s: concurrent tests' transient sessions land in the difference too and must be
|
||||||
|
// given time to finish and evict (every test closes what it creates).
|
||||||
|
const ATTEMPTS: usize = 600;
|
||||||
|
for attempt in 1..=ATTEMPTS {
|
||||||
|
let now: HashSet<String> = SESSIONS.lock().await.keys().cloned().collect();
|
||||||
|
let leftover: Vec<&String> = now.difference(&before).collect();
|
||||||
|
if leftover.is_empty() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Positional, not an inline `{leftover:?}`: on edition 2018 a lone-literal `panic!`
|
||||||
|
// does not go through format_args and would print the placeholder verbatim.
|
||||||
|
assert!(
|
||||||
|
attempt < ATTEMPTS,
|
||||||
|
"cancelled new() left a pc cached in SESSIONS (leftover: {:?})",
|
||||||
|
leftover
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The production shape of a controller session end: the offerer was created on a session's
|
||||||
|
// own current-thread runtime, the session queues its detached close and the runtime is
|
||||||
|
// destroyed immediately after (io_loop returning drops it). The pc's sockets and its
|
||||||
|
// ICE/DTLS/SCTP pump tasks must not die with that runtime, or the queued close completes
|
||||||
|
// without ever putting close_notify on the wire and the peer waits out ICE decay. Gathered
|
||||||
|
// (non-trickle) endpoints keep the cross-runtime signaling to two string handoffs.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn test_close_survives_creator_runtime_destruction() {
|
||||||
|
// tokio oneshots for every handoff: their sends are synchronous, and the session side
|
||||||
|
// awaits them INSIDE block_on — a current_thread runtime only drives its tasks while
|
||||||
|
// being block_on-driven, and the offerer's SCTP/ICE pumps must stay live until the
|
||||||
|
// answerer has confirmed the channel is open.
|
||||||
|
let (offer_tx, offer_rx) = tokio::sync::oneshot::channel::<String>();
|
||||||
|
let (answer_tx, answer_rx) = tokio::sync::oneshot::channel::<String>();
|
||||||
|
let (go_tx, go_rx) = tokio::sync::oneshot::channel::<()>();
|
||||||
|
let session = std::thread::spawn(move || {
|
||||||
|
let rt = tokio::runtime::Builder::new_current_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()
|
||||||
|
.expect("session runtime");
|
||||||
|
let offerer = rt.block_on(async {
|
||||||
|
let mut offerer = WebRTCStream::new("", false, 20000).await.unwrap();
|
||||||
|
offer_tx
|
||||||
|
.send(offerer.get_local_endpoint().await.unwrap())
|
||||||
|
.unwrap();
|
||||||
|
let answer = answer_rx.await.unwrap();
|
||||||
|
offerer.set_remote_endpoint(&answer).await.unwrap();
|
||||||
|
offerer.wait_connected(20000).await.unwrap();
|
||||||
|
// Keep driving the runtime until the answerer confirms open, so the EOF
|
||||||
|
// assertion — not connection setup — is what discriminates.
|
||||||
|
go_rx.await.unwrap();
|
||||||
|
offerer
|
||||||
|
});
|
||||||
|
// The session ends: queue the close, then destroy the runtime the pc was created
|
||||||
|
// on — exactly what io_loop returning does.
|
||||||
|
drop(crate::Stream::WebRTC(offerer));
|
||||||
|
drop(rt);
|
||||||
|
});
|
||||||
|
|
||||||
|
let offer = offer_rx.await.expect("offer from session thread");
|
||||||
|
let mut answerer = WebRTCStream::new(&offer, false, 20000).await.unwrap();
|
||||||
|
answer_tx
|
||||||
|
.send(answerer.get_local_endpoint().await.unwrap())
|
||||||
|
.expect("answer to session thread");
|
||||||
|
answerer.wait_connected(20000).await.unwrap();
|
||||||
|
go_tx.send(()).expect("release session thread");
|
||||||
|
session.join().expect("session thread");
|
||||||
|
|
||||||
|
match timeout(Duration::from_secs(5), answerer.next()).await {
|
||||||
|
Ok(None) | Ok(Some(Err(_))) => {}
|
||||||
|
Ok(Some(Ok(b))) => panic!("expected EOF after session close, got {} bytes", b.len()),
|
||||||
|
Err(_) => panic!("peer never observed the close after the creator runtime died"),
|
||||||
|
}
|
||||||
|
answerer.close().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The peer-initiated end: `next()` hits EOF and fires its own `close_detached` before the
|
||||||
|
// session's final close (here via `Stream`'s Drop) is issued. Both land on the serialized
|
||||||
|
// closer thread, so the first runs to completion and the second is a benign no-op — the pc
|
||||||
|
// must end up closed and evicted, never stranded by one closer cancelling the other.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn test_eof_close_then_drop_still_evicts() {
|
||||||
|
let (mut offerer, answerer) = connect_loopback().await;
|
||||||
|
let key = format!("offer:{}", offerer.session_key());
|
||||||
|
answerer.close().await;
|
||||||
|
match timeout(Duration::from_secs(5), offerer.next()).await {
|
||||||
|
Ok(None) | Ok(Some(Err(_))) => {} // EOF path fired close_detached internally
|
||||||
|
Ok(Some(Ok(b))) => panic!("expected EOF after peer close, got {} bytes", b.len()),
|
||||||
|
Err(_) => panic!("offerer.next() hung after peer close"),
|
||||||
|
}
|
||||||
|
drop(crate::Stream::WebRTC(offerer)); // second close via Drop
|
||||||
|
|
||||||
|
for _ in 0..200 {
|
||||||
|
if !SESSIONS.lock().await.contains_key(&key) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||||
|
}
|
||||||
|
panic!("pc still cached after EOF close + Drop close");
|
||||||
|
}
|
||||||
|
|
||||||
// Both framing violations must end the stream. The empty-FRAG_MORE case is the one that
|
// Both framing violations must end the stream. The empty-FRAG_MORE case is the one that
|
||||||
// cannot be caught downstream: it adds nothing to the accumulator, so the MAX_FRAME_LENGTH
|
// cannot be caught downstream: it adds nothing to the accumulator, so the MAX_FRAME_LENGTH
|
||||||
// cap never trips and `next()` would otherwise spin for as long as the peer keeps writing.
|
// cap never trips and `next()` would otherwise spin for as long as the peer keeps writing.
|
||||||
|
|||||||
Reference in New Issue
Block a user