Invented and developed several approaches for summarizing video by selecting appropriate keyframes and arranging them in layouts such as Manga comic books or stained-glass collages.
Technical challenges. Designed and contributed to several algorithms that perform video and audio analysis to segment a video and to select the number of keyframes needed by the application. Contributed to the initial Manga video summary invention and invented an improved layout algorithm that quickly fills all available space. Wrote versions using Java applets and timelines painted in HTML canvases filled combined with image placement in JavaScript. Implemented a client-server architecture that performed the layout on the server.
Technologies. Java, JavaScript, HTML Canvas, HTML Video.
Developed approaches for browsing through collections of videos and within a single video. Created several keyframe slider approaches that let users quickly locate interesting parts in videos. This work formed the basis of a Fuji Xerox product for managing lecture video collections.
Technical challenges. Contributed to analysis software for video segmentation and keyframe selection. Developed a web-based system that displays a file-system hierarchy of videos including a Java applet for each video that visualizes changes and other temporal data and controls a keyframe display via JavaScript. Created a helper application that launched a C++ app with an embedded Windows Media Player for playing the video from a shared file system (1998 technology) and controlling playback by skipping to interesting parts of the video. Created a video keyframe slider that uses a combination of HTML video and canvas to control video playback with keyframes attached to a timeline.
Technologies. C++, Windows Media, Microsoft SMB, Java, JavaScript, Common Gateway Interface (CGI).
Created algorithms and a search UI to successfully participate in the TRECVID evaluations.
Technical challenges. Contributed to the algorithms for analyzing the videos to be searched. Designed and developed a Java app that provides a UI for the search by visualizing matches in keyframe collages and in the timeline for the active video. Combined visual and text search.
Technologies. Java Swing, QuickTime, C++, DirectShow.