Source author record

Ruby Lee

Ruby Lee appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.

ResearcherUnclaimed source record

Artificial Intelligence Computation and Language Cryptography and Security eess.SP

Catalog footprint

What is connected

2works

4topics

4close collaborators

Actions

Connect this record

Open graph Browse works

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

preprint2021arXiv

ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

As mobile devices are becoming ubiquitous, regularly interacting with a variety of user interfaces (UIs) is a common aspect of daily life for many people. To improve the accessibility of these devices and to enable their usage in a variety of settings, building models that can assist users and accomplish tasks through the UI is vitally important. However, there are several challenges to achieve this. First, UI components of similar appearance can have different functionalities, making understanding their function more important than just analyzing their appearance. Second, domain-specific features like Document Object Model (DOM) in web pages and View Hierarchy (VH) in mobile applications provide important signals about the semantics of UI elements, but these features are not in a natural language format. Third, owing to a large diversity in UIs and absence of standard DOM or VH representations, building a UI understanding model with high coverage requires large amounts of training data. Inspired by the success of pre-training based approaches in NLP for tackling a variety of problems in a data-efficient way, we introduce a new pre-trained UI representation model called ActionBert. Our methodology is designed to leverage visual, linguistic and domain-specific features in user interaction traces to pre-train generic feature representations of UIs and their components. Our key intuition is that user actions, e.g., a sequence of clicks on different UI components, reveals important information about their functionality. We evaluate the proposed model on a wide variety of downstream tasks, ranging from icon classification to UI component retrieval based on its natural language description. Experiments show that the proposed ActionBert model outperforms multi-modal baselines across all downstream tasks by up to 15.5%.

preprint2020arXiv

Smartphone Impostor Detection with Built-in Sensors and Deep Learning

In this paper, we show that sensor-based impostor detection with deep learning can achieve excellent impostor detection accuracy at lower hardware cost compared to past work on sensor-based user authentication (the inverse problem) which used more conventional machine learning algorithms. While these methods use other smartphone users' sensor data to build the (user, non-user) classification models, we go further to show that using only the legitimate user's sensor data can still achieve very good accuracy while preserving the privacy of the user's sensor data (behavioral biometrics). For this use case, a key contribution is showing that the detection accuracy of a Recurrent Neural Network (RNN) deep learning model can be significantly improved by comparing prediction error distributions. This requires generating and comparing empirical probability distributions, which we show in an efficient hardware design. Another novel contribution is in the design of SID (Smartphone impostor Detection), a minimalist hardware accelerator that can be integrated into future smartphones for efficient impostor detection for different scenarios. Our SID module can implement many common Machine Learning and Deep Learning algorithms. SID is also scalable in parallelism and performance and easy to program. We show an FPGA prototype of SID, which can provide more than enough performance for real-time impostor detection, with very low hardware complexity and power consumption (one to two orders of magnitude less than related performance-oriented FPGA accelerators). We also show that the FPGA implementation of SID consumes 64.41X less energy than an implementation using the CPU with a GPU.