Muhammad Naufil

Developing data processing pipelines that transform raw camera and LiDAR ROS bag recordings into structured, ML-ready datasets for training autonomous driving models.

Working on understanding sign language and co-speech gestures through phonemes.

• Built a speech- and text-conditioned 3D gesture generation model with latent diffusion: a Transformer (DiT) denoises in latent space, conditioned on T5 text and mel-spectrogram audio features via cross-attention.
• Generates full-body SMPL-X motion (body, fingers, locomotion) at scale on the BEAT2, Embody3D and DnD datasets.
• Research questions: making characters spatially and socially aware, preventing collisions while keeping positions and orientations right for mutual gaze, encoding multi-participant speech cheaply, and scaling to any number of participants.

• Finetuned vision foundation models on a large-scale collection of deepfakes using clusters of 8×A100 GPUs.
• Generated deepfakes at scale with state-of-the-art open- and closed-source generative models to expand training coverage.
• Found that every generative model leaves a very different feature distribution, so the detector has to be finetuned on every new generator that comes out.

Evaluated instruction-following and few-shot learning of the latest open- and closed-source LLMs. Built a personal AI companion using GPT-4o with function calling and RAG retrieval (Pinecone). Explored agentic frameworks including n8n and crewAI to build AI workflows and automation pipelines.

Built up expertise in diffusion models progressively: started with unconditional image generation, then conditioned models on class labels to understand classifier-free guidance. Then moved to latent diffusion models for monadic co-speech gesture synthesis, first unconditional and then conditioned on speech transcription to generate gestures that naturally accompany spoken language.

Developed a one-shot computer vision system for retail shelf monitoring. Evaluated SuperGlue and LightGlue to recognize products from a single reference image and detect out-of-stock items, so the model doesn't need retraining every time packaging changes.

Markerless 3D motion capture from ordinary phone cameras for ergonomic safety analytics on the factory floor.

• Built the MoCap product with a single colleague, winning a $150K/year contract with Siemens, and landed 3M as the company's first customer.
• Experimented with monocular and multi-view approaches to extract 3D joint locations of workers; tackled monocular ambiguity with limb-length constraints and temporal smoothing.
• Extended an open-source monocular synthetic 3D motion pipeline to multi-view in Blender (camera intrinsics, extrinsics, world and camera coordinates), and synthesized human motion in an anthropomorphic simulation for detailed ergonomics studies.
• Built a mini MoCap system from two RGB-D cameras with confidence-based fusion.

Mistake-proofing for manual assembly. It tracks each step of an assembly process and gives audible and visual alerts to help workers avoid mistakes.

• Cut purchasing price by 150× and deployment time by 180× for a discrete manufacturing customer by replacing their sensor and visual inspection setup with a human action recognition model on Jetson Xavier / AGX.
• Built the human activity segmentation demo for the SBIR grant from NASA's Human Research Program.
• Benchmarked video understanding models (3D CNN, I3D) against plain object detection (YOLO) and found that simple object detection works better for recognizing assembly steps on the factory floor.

Collects timing data for every cycle on an assembly line, flags non-value-add work, finds the most efficient cycles and grades each session, so industrial engineers can improve their processes.

• Wrote the algorithm for time and motion studies on discrete manufacturing assembly, giving step-level analytics.
• Ran pilot projects with Honda, Suzuki and Nissan. Supervised the data collection and annotation team and kept improving the models to handle different factory floors.