A research program at AutismAILab

HEARTH

Home Environment Ambient sensing for Real-Time Health

HEARTH is a research program for passive, contact-free, in-home health monitoring of older adults and persons with disabilities. It rejects the trade-off that current home monitoring asks of vulnerable people: wear a device, accept a camera, step on a mat in order to be cared for.

The signals required to characterize a person's movement, gait, sleep, falls, and cardiorespiratory state are already present in the home environment. Modern machine learning is sufficient to recover them without adding any burden to the monitored individual. The cost should fall on the engineering, not on the person.

Two complementary modalities

WAVE

WiFi Ambient Vitality and Event-detection

Recovers motion, gait, occupancy, falls, and coarse respiratory rate from the perturbations a body imposes on the multipath structure of ordinary WiFi signals. No camera, no wearable.

PULSE

Passive Unobtrusive Light-based Signal Extraction

Recovers heart rate, heart rate variability, respiratory rate from the chromatic modulation each heartbeat imposes on the visible color of skin, observed with an ordinary RGB camera.

Their blind spots are complementary. WAVE penetrates walls but cannot resolve a heartbeat in an uncontrolled home. PULSE resolves the heartbeat exquisitely but requires the person in frame with adequate light. Operated together, the caregiver app can prefer PULSE when it is available, fall back to WAVE-derived respiratory rate when it is not, and continuously monitor movement regardless.

WiFi sensing modality

WAVE

Concept

WAVE recovers human motion from the perturbations a body imposes on the multipath structure of WiFi signals. Every modern WiFi link carries, alongside its data, a continuous estimate of the wireless channel between transmitter and receiver, known as the Channel State Information, or CSI. CSI is sensitive to the geometry and motion of every scattering object in the link's environment, the human body included.

The vision: a system that operates in the background of ordinary home networking, captures motion at room scale, and surfaces only what the caregiver actually needs. The moments and the trends that matter, not a granular log of activity.

The science

The body as scatterer. At WiFi frequencies (2.4, 5, 6 GHz), wavelengths are 5–12 cm, placing limbs and torsos in the Mie scattering regime. Walking creates a torso Doppler around ±1 Hz with limb sidebands at ±2 to 4 Hz; sitting still leaves only the slow Doppler of breathing in the ±0.1 to 1 Hz range; falls produce broad-band Doppler events with distinctive temporal signatures.

The inverse problem. Recovering body configuration from CSI is ill-posed in general. The practical solution is the cross-modal supervision paradigm: train a neural network with paired CSI and synchronized RGB video, using a vision-based pose estimator as pseudo-ground-truth. At inference, only CSI is needed. This is the technique behind Person-in-WiFi, DensePose-from-WiFi, GoPose, and VST-Pose.

What WAVE can detect

• Coarse activity: sitting, standing, walking, lying, transitioning. Within-environment 95–99%; cross-environment 60–80% without adaptation.
• Falls: 94–99% in controlled trials. Generalization to elderly fall kinematics largely unvalidated; a real opportunity.
• Gait biomarkers: gait speed, gait variability, sit-to-stand cadence. Already clinically validated as biomarkers of fall risk, frailty, and early cognitive decline.
• Occupancy and room-level trajectory: nighttime ambulation, sustained-immobility detection.
• Respiratory rate: ±2 bpm under reasonable conditions; validated in older adults at home (Alzaabi et al., 2025).
• Heart rate: substantially noisier in uncontrolled homes. This is why PULSE exists.

Hardware

WAVE uses commodity hardware: four ESP32-S3 modules as beacon transmitters (50–200 pps), paired with a Raspberry Pi 5 running Nexmon-patched CSI capture on its Broadcom radio. The Pi performs preprocessing, on-device inference for low-latency events such as falls, and feature extraction. Only learned features and event-level summaries leave the home; raw CSI never does. The architecture is forward-compatible with IEEE 802.11bf, the WLAN sensing amendment ratified October 2024.

Research aims

Cross-environment, cross-subject generalization with few-shot personalization. A small, well-documented multi-home study with a 1–5 minute calibration protocol against held-out homes.
Longitudinal trajectory biomarkers. Almost all published CSI work treats activities as snippets. Caregiving needs trends. The Volitiq statistical machinery (GAMM trajectories, GP normative modeling) applies directly.
Multi-user disambiguation in real homes. Practically essential because the caregiver is also in the home.
Edge deployment with federated personalization. On-device inference paired with federated learning over only the model adapter weights, achieving privacy and cross-environment generalization through a single mechanism.
Disability-population-specific kinematics. No public dataset adequately represents the gait of people with mobility disabilities, post-stroke conditions, Parkinson’s, or muscular dystrophy.

Papers in flight

Paper W1

"Few-shot Personalization for Cross-Home WiFi-CSI Activity Recognition: A Multi-Household Study"

ACM IMWUT · IEEE TMC · IEEE IoT-J
Paper W2

"A Caregiver-Oriented Passive Sensing Platform for In-Home Movement Monitoring: Design, Deployment, and Field Evaluation"

IEEE JBHI (primary) · ACM CHI
Paper W3

"Longitudinal WiFi-Derived Gait Biomarkers Track Standard Mobility Assessments in Community-Dwelling Older Adults"

npj Digital Medicine · J. NeuroEng. Rehab.

RGB rPPG modality

PULSE

Concept

PULSE recovers a person's cardiovascular and respiratory state from the subtle chromatic modulation that each heartbeat imposes on the visible color of their skin. The phenomenon is invisible to the naked eye but routinely measurable from ordinary RGB video with appropriate signal processing.

The vision: a single, fixed RGB camera positioned somewhere the monitored person is likely to be seated, such as a favorite chair, a reading area, or a kitchen table. When the person enters the frame, PULSE extracts a continuous pulse waveform from their face or hand, derives heart rate and HRV, estimates respiratory rate, and contributes these signals to the unified HEARTH caregiver record. When the person is not in frame, PULSE is silent.

The science

Photoplethysmography. Each heartbeat sends a transient bolus of hemoglobin-laden blood into the cutaneous microcirculation. Oxygenated hemoglobin has a strong absorption peak around 540 nm in the green band. Each pulse causes a small but predictable modulation of the skin's apparent color (typically 0.5 to 2% of mean skin reflectance) that sits below the threshold of human perception but remains recoverable from video given adequate temporal stability and reliable face tracking.

Three families of methods anchor the field:

• Blind source separation (Poh, McDuff, Picard 2010): ICA on the three RGB channels. Effective in clean conditions but brittle under motion.
• Chrominance projection (CHROM) (De Haan & Jeanne 2013): constructs chrominance signals that change differentially under pulse vs. motion. A major step in motion robustness.
• Plane Orthogonal to Skin (POS) (Wang et al., 2017): projects RGB onto the plane orthogonal to the skin-tone vector. Strong handcrafted baseline against which deep methods are evaluated.

Modern deep methods. Since DeepPhys (Chen & McDuff, 2018), the field has shifted to end-to-end learning. State of the art now includes PhysFormer (transformer with temporal difference), CodePhys (discrete codebook learned from contact-sensor data, Feb 2025), TYrPPG (Mamba-based, Nov 2025), and Spiking-PhysFormer (spiking NN for energy-efficient edge deployment, 2025).

What PULSE can detect

• Heart rate: ±1–3 bpm under reasonable conditions, comparable to mid-grade contact sensors.
• Heart rate variability: feasible under stationary conditions. HRV (RMSSD, pNN50, LF/HF power) is clinically more informative than mean HR for autonomic function.
• Respiratory rate: extractable from breath-driven modulation of the pulse waveform, ±2 bpm.
• Sleep & autonomic indicators: vagal-mediated HR/HRV changes during sleep; longitudinal baseline HRV informative about chronic stress.

Hardware

A single fixed RGB camera (60 fps, global shutter where possible) with a small embedded GPU (NVIDIA Jetson Nano-class) for on-device inference. Face detection, tracking, pulse extraction, and HR estimation all run locally. A hardware shutter, visible to the monitored person and physically obvious, ensures they can opt out at any time without depending on a software toggle. Only extracted vital-sign features leave the home, never raw video.

Research aims

Generalization across skin tone, age, and disability. Almost all published benchmarks skew young, neurotypical, and lightly pigmented.
Robustness in real homes versus lab conditions. Lighting variability, partial face occlusion, resting postures other than upright.
Heart rate variability beyond mean HR. Beat-level accuracy is harder than mean HR but clinically more informative for autonomic function.
Multi-modal fusion with WAVE. Joint estimation of vital signs and motion biomarkers is essentially untouched in the literature.
Edge inference at low power. Spiking-PhysFormer-style models running on a small SoC near the camera.

Papers in flight

Paper P1

"Robust Heart Rate Variability Estimation from Remote Photoplethysmography in Older Adults: A CodePhys-Inspired Adaptation"

IEEE T-BME · IEEE JBHI · Physiological Measurement
Paper P2

"Multi-Modal Passive Sensing in the Home: Joint WiFi-CSI and rPPG for Continuous Caregiver-Oriented Monitoring"

IEEE JBHI · ACM IMWUT · Nature Scientific Reports
Paper P3

"Longitudinal rPPG Vital-Sign Biomarkers in Community-Dwelling Older Adults: A 30-Day In-Home Validation Study"

npj Digital Medicine · JMIR mHealth & uHealth

Consolidated 24-month plan

Roadmap

Phase	Months	WAVE focus	PULSE focus
01	1–2	ESP32-S3 + Pi 5 rig. Reproduce esp-csi. First lab dataset.	Camera rig. Implement CHROM and POS. First lab dataset against pulse-oximeter ground truth.
02	3–4	CNN-LSTM and Attention-GRU baselines on UT-HAR, Widar 3.0. Cross-environment gap quantified.	Reproduce DeepPhys and PhysFormer on COHFACE, UBFC-rPPG. Cross-dataset gap quantified.
03	5–9	Multi-home few-shot personalization study. Paper W1.	HRV-focused adaptation for older adults. Paper P1.
04	10–15	Multi-home pilot under IRB protocol (subject to approval). Caregiver dashboard. Paper W2.	Co-deployment with WAVE. Multi-modal fusion. Paper P2 (joint with W2).
05	16–24+	Longitudinal correlation with TUG and clinical mobility assessments. Paper W3.	30-day in-home validation against contact gold standards. Paper P3.

How we operate

Ethical and regulatory posture

HEARTH is being designed from the start under three principles that flow from its disability-ethics framing.

Visibility

The monitored person sees what the caregiver sees.

Control

The monitored person controls when sensing pauses.

Trends, not logs

The system surfaces trends and deviations, not granular activity logs.

Human-subjects research

Any human-subjects deployment will go through IRB review at an affiliated institution before data collection begins. Protocol design, including consent procedures appropriate to the population and proxy-consent provisions where legally authorized representatives are involved, is part of the program planning. No approvals are in place yet; we expect this to be developed alongside the clinical partners who join the program.

Clinical-partner data sharing

If and when physiological signals leave the lab and enter a clinical care setting, the regulatory frame for that setting (HIPAA in the US, equivalents elsewhere) will apply. The architecture is being designed with that boundary in mind so that the operational deployment can meet the requirement once the clinical partnership context is concrete.

Device / diagnostic classification

HEARTH is a research instrument, not a medical device. Any future diagnostic or clinical-decision claim (for example a heart-rate or fall-risk indication intended to inform care) would require evaluation against the appropriate regulatory pathway, decided with clinical partners and only if the underlying data supports the claim.

Privacy by construction

The design intent is that raw video and raw WiFi channel data never leave the home. Only learned features and event-level summaries are transmitted, and storage/retention are scoped to what the analysis actually requires. The full data-handling story will be specified per-deployment alongside the IRB protocol.

Team & affiliations

Who is building this

Shailesh Pandey

PhD candidate (ABD), EECS, University of Kansas

Lead researcher and architect of the HEARTH program. Founder, AutismAILab. Dissertation advisors: Dr. Sumaiya Shomaji and Dr. Matt Mosconi.

Institutional partners

• University of Kansas, EECS: academic home
• K-CART: Kansas Center for Autism Research and Training; neurodevelopmental recruitment pathway
• ITTC: Information and Telecommunication Technology Center; infrastructure access
• AutismAILab: operating entity for the productization track

Open collaborator positions

HEARTH is actively seeking collaborators in geriatric care, occupational therapy, gerontechnology, and disability advocacy. Interested clinicians, researchers, and self-advocates are invited to make contact.

Get in touch →

Working document, maintained by the lead researcher. Comments and collaboration inquiries welcome. Last updated May 2026.

← Back to apps