Showing papers by "Shi-Min Hu published in 2015"

3D indoor scene modeling from RGB-D data: a survey

Abstract: 3D scene modeling has long been a fundamental problem in computer graphics and computer vision. With the popularity of consumer-level RGB-D cameras, there is a growing interest in digitizing real-world indoor 3D scenes. However, modeling indoor 3D scenes remains a challenging problem because of the complex structure of interior objects and poor quality of RGB-D data acquired by consumer-level sensors. Various methods have been proposed to tackle these challenges. In this survey, we provide an overview of recent advances in indoor scene modeling techniques, as well as public datasets and code libraries which can facilitate experiments and evaluation.

PatchTable: efficient patch queries for large datasets and applications

Abstract: This paper presents a data structure that reduces approximate nearest neighbor query times for image patches in large datasets. Previous work in texture synthesis has demonstrated real-time synthesis from small exemplar textures. However, high performance has proved elusive for modern patch-based optimization techniques which frequently use many exemplar images in the tens of megapixels or above. Our new algorithm, PatchTable, offloads as much of the computation as possible to a pre-computation stage that takes modest time, so patch queries can be as efficient as possible. There are three key insights behind our algorithm: (1) a lookup table similar to locality sensitive hashing can be precomputed, and used to seed sufficiently good initial patch correspondences during querying, (2) missing entries in the table can be filled during pre-computation with our fast Voronoi transform, and (3) the initially seeded correspondences can be improved with a precomputed k-nearest neighbors mapping. We show experimentally that this accelerates the patch query operation by up to 9× over k-coherence, up to 12× over TreeCANN, and up to 200× over PatchMatch. Our fast algorithm allows us to explore efficient and practical imaging and computational photography applications. We show results for artistic video stylization, light field super-resolution, and multi-image editing.

Magic decorator: automatic material suggestion for indoor digital scenes

Abstract: Assigning textures and materials within 3D scenes is a tedious and labor-intensive task. In this paper, we present Magic Decorator, a system that automatically generates material suggestions for 3D indoor scenes. To achieve this goal, we introduce local material rules, which describe typical material patterns for a small group of objects or parts, and global aesthetic rules, which account for the harmony among the entire set of colors in a specific scene. Both rules are obtained from collections of indoor scene images. We cast the problem of material suggestion as a combinatorial optimization considering both local material and global aesthetic rules. We have tested our system on various complex indoor scenes. A user study indicates that our system can automatically and efficiently produce a series of visually plausible material suggestions which are comparable to those produced by artists.

Cracking App Isolation on Apple: Unauthorized Cross-App Resource Access on MAC OS~X and iOS

Abstract: On modern operating systems, applications under the same user are separated from each other, for the purpose of protecting them against malware and compromised programs. Given the complexity of today's OSes, less clear is whether such isolation is effective against different kind of cross-app resource access attacks (called XARA in our research). To better understand the problem, on the less-studied Apple platforms, we conducted a systematic security analysis on MAC OS~X and iOS. Our research leads to the discovery of a series of high-impact security weaknesses, which enable a sandboxed malicious app, approved by the Apple Stores, to gain unauthorized access to other apps' sensitive data. More specifically, we found that the inter-app interaction services, including the keychain, WebSocket and NSConnection on OS~X and URL Scheme on the MAC OS and iOS, can all be exploited by the malware to steal such confidential information as the passwords for iCloud, email and bank, and the secret token of Evernote. Further, the design of the app sandbox on OS~X was found to be vulnerable, exposing an app's private directory to the sandboxed malware that hijacks its Apple Bundle ID. As a result, sensitive user data, like the notes and user contacts under Evernote and photos under WeChat, have all been disclosed. Fundamentally, these problems are caused by the lack of app-to-app and app-to-OS authentications. To better understand their impacts, we developed a scanner that automatically analyzes the binaries of MAC OS and iOS apps to determine whether proper protection is missing in their code. Running it on hundreds of binaries, we confirmed the pervasiveness of the weaknesses among high-impact Apple apps. Since the issues may not be easily fixed, we built a simple program that detects exploit attempts on OS~X, helping protect vulnerable apps before the problems can be fully addressed.

Fast multiple-fluid simulation using Helmholtz free energy

Abstract: Multiple-fluid interaction is an interesting and common visual phenomenon we often observe. In this paper, we present an energy-based Lagrangian method that expands the capability of existing multiple-fluid methods to handle various phenomena, such as extraction, partial dissolution, etc. Based on our user-adjusted Helmholtz free energy functions, the simulated fluid evolves from high-energy states to low-energy states, allowing flexible capture of various mixing and unmixing processes. We also extend the original Cahn-Hilliard equation to be better able to simulate complex fluid-fluid interaction and rich visual phenomena such as motion-related mixing and position based pattern. Our approach is easily integrated with existing state-of-the-art smooth particle hydrodynamic (SPH) solvers and can be further implemented on top of the position based dynamics (PBD) method, improving the stability and incompressibility of the fluid during Lagrangian simulation under large time steps. Performance analysis shows that our method is at least 4 times faster than the state-of-the-art multiple-fluid method. Examples are provided to demonstrate the new capability and effectiveness of our approach.

Simultaneous Camera Path Optimization and Distraction Removal for Improving Amateur Video

Abstract: A major difference between amateur and professional video lies in the quality of camera paths. Previous work on video stabilization has considered how to improve amateur video by smoothing the camera path. In this paper, we show that additional changes to the camera path can further improve video aesthetics. Our new optimization method achieves multiple simultaneous goals: 1) stabilizing video content over short time scales; 2) ensuring simple and consistent camera paths over longer time scales; and 3) improving scene composition by automatically removing distractions, a common occurrence in amateur video. Our approach uses an $L^{1}$ camera path optimization framework, extended to handle multiple constraints. Two passes of optimization are used to address both low-level and high-level constraints on the camera path. The experimental and user study results show that our approach outputs video that is perceptually better than the input, or the results of using stabilization only.

Active Exploration of Large 3D Model Repositories

Abstract: With broader availability of large-scale 3D model repositories, the need for efficient and effective exploration becomes more and more urgent. Existing model retrieval techniques do not scale well with the size of the database since often a large number of very similar objects are returned for a query, and the possibilities to refine the search are quite limited. We propose an interactive approach where the user feeds an active learning procedure by labeling either entire models or parts of them as “like” or “dislike” such that the system can automatically update an active set of recommended models. To provide an intuitive user interface, candidate models are presented based on their estimated relevance for the current query. From the methodological point of view, our main contribution is to exploit not only the similarity between a query and the database models but also the similarities among the database models themselves. We achieve this by an offline pre-processing stage, where global and local shape descriptors are computed for each model and a sparse distance metric is derived that can be evaluated efficiently even for very large databases. We demonstrate the effectiveness of our method by interactively exploring a repository containing over 100 K models.

A response time model for abrupt changes in binocular disparity

Abstract: We propose a novel depth perception model to determine the time taken by the human visual system (HVS) to adapt to an abrupt change in stereoscopic disparity, such as can occur in a scene cut. A series of carefully designed perceptual experiments on successive disparity contrast were used to build our model. Factors such as disparity, changes in disparity, and the spatial frequency of luminance contrast were taken into account. We further give a computational method to predict the response time during scene cuts in stereoscopic cinematography, which has been validated in user studies. We also consider various applications of our model.

Panorama completion for street views

Abstract: This paper considers panorama images used for street views. Their viewing angle of 360° causes pixels at the top and bottom to appear stretched and warped. Although current image completion algorithms work well, they cannot be directly used in the presence of such distortions found in panoramas of street views. We thus propose a novel approach to complete such 360° panoramas using optimization-based projection to deal with distortions. Experimental results show that our approach is efficient and provides an improvement over standard image completion algorithms.

Unauthorized Cross-App Resource Access on MAC OS X and iOS.

Abstract: On modern operating systems, applications under the same user are separated from each other, for the purpose of protecting them against malware and compromised programs. Given the complexity of today's OSes, less clear is whether such isolation is effective against different kind of cross-app resource access attacks (called XARA in our research). To better understand the problem, on the less-studied Apple platforms, we conducted a systematic security analysis on MAC OS~X and iOS. Our research leads to the discovery of a series of high-impact security weaknesses, which enable a sandboxed malicious app, approved by the Apple Stores, to gain unauthorized access to other apps' sensitive data. More specifically, we found that the inter-app interaction services, including the keychain, WebSocket and NSConnection on OS~X and URL Scheme on the MAC OS and iOS, can all be exploited by the malware to steal such confidential information as the passwords for iCloud, email and bank, and the secret token of Evernote. Further, the design of the app sandbox on OS~X was found to be vulnerable, exposing an app's private directory to the sandboxed malware that hijacks its Apple Bundle ID. As a result, sensitive user data, like the notes and user contacts under Evernote and photos under WeChat, have all been disclosed. Fundamentally, these problems are caused by the lack of app-to-app and app-to-OS authentications. To better understand their impacts, we developed a scanner that automatically analyzes the binaries of MAC OS and iOS apps to determine whether proper protection is missing in their code. Running it on hundreds of binaries, we confirmed the pervasiveness of the weaknesses among high-impact Apple apps. Since the issues may not be easily fixed, we built a simple program that detects exploit attempts on OS~X, helping protect vulnerable apps before the problems can be fully addressed.

Pairminer: mining for paired functions in Kernel extensions

Abstract: Drivers use kernel extension functions to manage devices, and there are often many rules on how they should be used. Among the rules, utilization of paired functions, which means that the functions must be called in pairs between two different functions, is extremely complex and important. However, such pairing rules are not well documented, and these rules can be easily violated by programmers when they unconsciously ignore or forget about them. Therefore it is useful to develop a tool to automatically extract paired functions in the kernel source and detect incorrect usages. We put forward a method called PairMiner in this paper. Heuristic and statistical mechanisms are adopted to associate with the special structure of drivers’ source code, to find out paired functions between relative operations, and then to detect violations with extracted paired functions. In the experiment evaluation, we have successfully found 1023 paired functions in Linux 3.10.10. The utility of PairMiner was evaluated by analyzing the source code of Linux 2.6.38 and 3.10.10. PairMiner located 265 bugs about paired function violations in 2.6.38 which have been fixed in 3.10.10. We also have identified 1994 paired function violations which have not yet been fixed in 3.10.10. We have reported some violations as potential bugs with emails to the developers, 27 developers have replied the emails and 20 bugs have been confirmed so far, 2 violations are confirmed as false positive.

Automated resource release in device drivers

Abstract: Device drivers require system resources to control hardware and provide fundamental services for applications. The acquired resources must be explicitly released by drivers. Otherwise, these resources will never be reclaimed by the operating system, and they are not available for other programs any more, causing hard-to-find system problems. We study on Linux driver mailing lists, and find many applied patches handle improper resource-release operations, especially in error handling paths. In order to improve current resource management and avoid resource-release omissions in device drivers, we propose a novel approach named AutoRR, which can automatically and safely release resources based on specification-mining techniques. During execution, we maintain a resource-state table by recording the runtime information of function calls. If the driver fails to release acquired resources during execution, AutoRR will report violations and call corresponding releasing functions with the recorded runtime information to release acquired resources. To fully and safely release acquired resources, a dynamic analysis of resource dependency and allocation hierarchy is also performed, which can avoid dead resources and double frees. AutoRR works in both normal execution and error handling paths for reliable resource management. We implement AutoRR with LLVM, and evaluate it on 8 Ethernet drivers in Linux 3.17.2. The evaluation shows that the overhead of AutoRR is very low, and it has successfully fixed 18 detected resource-release omission violations without side effects. Our work shows a feasible way of using specification-mining results to avoid related violations.

WebC: toward a portable framework for deploying legacy code in web browsers

Abstract: For security, most web applications are developed in some type-safe language, such as JavaScriptor Java. However, there is a huge amount of legacy codes developed in unsafe languages, which provide richfunctionality and are more efficient than their type-safe counterparts. To allow browsers to incorporate type-safecomponents in a secure way, previous approaches use the software-based fault isolation (SFI) to isolate untrustedlegacy code. The SFI approach performs machine-code transformation for security, but the downside is the lossof architecture independence. We propose WebC, a system that allows legacy code transmitted over the web viathe Low Level Virtual Machine (LLVM) bitcode format. The untrusted bitcode is transformed by WebC intocode in the WebC security language, which enforces both memory isolation and control-flow integrity. Comparedwith previous approaches, WebC is more portable, provides stronger security, and allows more flexible memorymanagement. Experimental results show that the average runtime overhead of WebC is modest.

Complete Runtime Tracing for Device Drivers Based on LLVM

Abstract: Device drivers often suffer from much more bugs than the kernel, so testing device drivers becomes more and more important and necessary. In software testing, runtime tracing is an important technique to monitor real executing procedures of the program. Meanwhile, runtime information can also assist the programmer to make more accurate analysis of the program, like verifying the correctness of code execution and detecting bugs. However, due to kernel-mode execution and high complexity of kernel code, completely tracing drivers is hard, which causes real execution paths can not be clearly identified. In order to provide more powerful support for software testing of device drivers, we propose a method named Driver Trace, to do complete runtime tracing at the function level. Driver Trace utilizes instrumentation technique for runtime tracing, which is implemented based on LLVM compiler infrastructure. When the target driver works, Driver Trace records complete runtime information of function calls, like function names, return values and parameter pointers, and the information is recorded in a log file for future analysis. We have successfully implemented Driver Trace on 10 real device drivers in Linux 3.16.4 and made the evaluation as well. The experimental results show that Driver Trace provides an effective method of runtime tracing for device drivers with the modest overhead. Moreover, using an automated analysis of the runtime information recorded by Driver Trace, we also find 6 violations about resource usages in these 10 device drivers.

Patchnet model and building method therefor

Abstract: The present invention relates to the technical field of image processing, and in particular, to a patch net model for image representation and a construction method of the patch net model. The patch net model for image representation is of a forest-shaped structure consisting of a plurality of composite nodes and basic nodes, each composite node is a non-leaf node, and each basic node is a leaf node; the basic node includes a certain patch region of an image and a representative patch representing an apparent feature of the patch region; the composite node includes a certain patch region of the image and can be further decomposed into basic nodes and/or composite nodes; an edge exists between two nodes, which are located on the same layer of the forest-shaped structure and are spatially connected, and a relation matrix used for expressing the spatial relative position of the two nodes is arranged on the edge.