BZPEERReading, trust and collaboration for research
Menu

Discover

GraphFollow connections around a paper, topic or saved view.

Workspaces

HomePick up where your research flow left off.InboxHandle requests, replies and notifications from one place.CollectionsCurate reading lists, evidence sets and shareable dossiers.ResearchSee your private provenance graph, trail and imports.CollaborationMove from discovery into focused discussion rooms.PublishDraft, preview and publish full-text research articles.

Network

ExpertsFind specialists worth following or asking for help.CommunitiesJoin research circles organized around shared work.PartnersSee external organizations active around the same topics.

Opportunities

ListingsBrowse roles, calls and collaborations with clearer fit signals.

Account

Log inReturn to your reading and collaboration workspace.Create accountStart saving work, following people and joining teams.
Log inCreate account

Researcher profile

Tao Xiang

Tao Xiang contributes to research discovery and scholarly infrastructure.

ResearcherAffiliation not importedOpen to collaborate
Computer Visioncond-mat.supr-conMachine Learningcond-mat.str-elArtificial Intelligencecond-mat.mtrl-sciMultimediacond-mat.mes-hallphysics.comp-phComputation and LanguageCryptography and Securityeess.IVeess.SPeess.SYGraphicshep-thInformation Retrievalmath-phmath.MPNeural and Evolutionary Computingquant-phSystems and Control

Trust snapshot

Quick read

Trust 21 - EmergingVerification L1Unclaimed author
57works
0followers
22topics
4close collaborators

Actions

Decide how to stay connected

Follow researcher0

Identity and collaboration

How to connect with this researcher

Claiming links this public author record to a researcher profile and unlocks direct collaboration workflows.

Log in to claim

Direct collaboration

Open a focused conversation when the fit is right

Claim this author entity first to unlock direct invitations.

Research graph

See the researcher in context

Open full explorer

Inspect adjacent work, topics, institutions and collaborators without jumping out to a separate graph page.

Building this graph slice

BZPEER is loading the nearby papers, people, topics and institutions for this page.

Published work

57 published item(s)

preprint2026arXiv

From Ground to Sky: Architectures, Applications, and Challenges Shaping Low-Altitude Wireless Networks

In this article, we introduce a novel low-altitude wireless network (LAWN), which is a reconfigurable, three-dimensional (3D) layered architecture. In particular, the LAWN integrates connectivity, sensing, control, and computing across aerial and terrestrial nodes that enable seamless operation in complex, dynamic, and mission-critical environments. Different from the conventional aerial communication systems, LAWN's distinctive feature is its tight integration of functional planes in which multiple functionalities continually reshape themselves to operate safely and efficiently in the low-altitude sky. With the LAWN, we discuss several enabling technologies, such as integrated sensing and communication (ISAC), semantic communication, and fully-actuated control systems. Finally, we identify potential applications and key cross-layer challenges. This article offers a comprehensive roadmap for future research and development in the low-altitude airspace.

0 citations0 reviewsNo code linkedOpen access
preprint2024arXiv

Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks

Solving image-to-3D from a single view is an ill-posed problem, and current neural reconstruction methods addressing it through diffusion models still rely on scene-specific optimization, constraining their generalization capability. To overcome the limitations of existing approaches regarding generalization and consistency, we introduce a novel neural rendering technique. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Specifically, our method builds neural encoding volumes from generated multi-view inputs. We adjust the weights of the SDF network conditioned on an input image at test-time to allow model adaptation to novel scenes in a feed-forward manner via HyperNetworks. To mitigate artifacts derived from the synthesized views, we propose the use of a volume transformer module to improve the aggregation of image features instead of processing each viewpoint separately. Through our proposed method, dubbed as Hyper-VolTran, we avoid the bottleneck of scene-specific optimization and maintain consistency across the images generated from multiple viewpoints. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Adaptive Fine-Grained Sketch-Based Image Retrieval

The recent focus on Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) has shifted towards generalising a model to new categories without any training data from them. In real-world applications, however, a trained FG-SBIR model is often applied to both new categories and different human sketchers, i.e., different drawing styles. Although this complicates the generalisation problem, fortunately, a handful of examples are typically available, enabling the model to adapt to the new category/style. In this paper, we offer a novel perspective -- instead of asking for a model that generalises, we advocate for one that quickly adapts, with just very few samples during testing (in a few-shot manner). To solve this new problem, we introduce a novel model-agnostic meta-learning (MAML) based framework with several key modifications: (1) As a retrieval task with a margin-based contrastive loss, we simplify the MAML training in the inner loop to make it more stable and tractable. (2) The margin in our contrastive loss is also meta-learned with the rest of the model. (3) Three additional regularisation losses are introduced in the outer loop, to make the meta-learned FG-SBIR model more effective for category/style adaptation. Extensive experiments on public datasets suggest a large gain over generalisation and zero-shot based approaches, and a few strong few-shot baselines.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Chiral conformal field theory for topological states and the anyon eigenbasis on the torus

Model wave functions constructed from (1+1)D conformal field theory (CFT) have played a vital role in studying chiral topologically ordered systems. There usually exist multiple degenerate ground states when such states are placed on the torus. The common practice for dealing with this degeneracy within the CFT framework is to take a full correlator on the torus, which includes both holomorphic and antiholomorphic sectors, and decompose it into several conformal blocks. In this paper, we propose a pure chiral approach for the torus wave function construction. By utilizing the operator formalism, the wave functions are written as chiral correlators of holormorphic fields restricted to each individual topological sector. This method is not only conceptually much simpler, but also automatically provides us the anyon eigenbasis of the degenerate ground states (also known as the "minimally entangled states"). As concrete examples, we construct the full set of degenerate ground states for SO($n$)$_1$ and SU($n$)$_1$ chiral spin liquids on the torus, the former of which provide a complete wave function realization of Kitaev's sixteenfold way of anyon theories. We further characterize their topological orders by analytically computing the associated modular $S$ and $T$ matrices.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Deep Learning for Free-Hand Sketch: A Survey

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Domain Generalization: A Survey

Generalization to out-of-distribution (OOD) data is a capability natural to humans yet challenging for machines to reproduce. This is because most learning algorithms strongly rely on the i.i.d.~assumption on source/target data, which is often violated in practice due to domain shift. Domain generalization (DG) aims to achieve OOD generalization by using only source data for model learning. Over the last ten years, research in DG has made great progress, leading to a broad spectrum of methodologies, e.g., those based on domain alignment, meta-learning, data augmentation, or ensemble learning, to name a few; DG has also been studied in various application areas including computer vision, speech recognition, natural language processing, medical imaging, and reinforcement learning. In this paper, for the first time a comprehensive literature review in DG is provided to summarize the developments over the past decade. Specifically, we first cover the background by formally defining DG and relating it to other relevant fields like domain adaptation and transfer learning. Then, we conduct a thorough review into existing methods and theories. Finally, we conclude this survey with insights and discussions on future research directions.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches

The human visual system is remarkable in learning new visual concepts from just a few examples. This is precisely the goal behind few-shot class incremental learning (FSCIL), where the emphasis is additionally placed on ensuring the model does not suffer from "forgetting". In this paper, we push the boundary further for FSCIL by addressing two key questions that bottleneck its ubiquitous application (i) can the model learn from diverse modalities other than just photo (as humans do), and (ii) what if photos are not readily accessible (due to ethical and privacy constraints). Our key innovation lies in advocating the use of sketches as a new modality for class support. The product is a "Doodle It Yourself" (DIY) FSCIL framework where the users can freely sketch a few examples of a novel class for the model to learn to recognize photos of that class. For that, we present a framework that infuses (i) gradient consensus for domain invariant learning, (ii) knowledge distillation for preserving old class information, and (iii) graph attention networks for message passing between old and novel classes. We experimentally show that sketches are better class support than text in the context of FSCIL, echoing findings elsewhere in the sketching literature.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Dynamic Instance Domain Adaptation

Most existing studies on unsupervised domain adaptation (UDA) assume that each domain's training samples come with domain labels (e.g., painting, photo). Samples from each domain are assumed to follow the same distribution and the domain labels are exploited to learn domain-invariant features via feature alignment. However, such an assumption often does not hold true -- there often exist numerous finer-grained domains (e.g., dozens of modern painting styles have been developed, each differing dramatically from those of the classic styles). Therefore, forcing feature distribution alignment across each artificially-defined and coarse-grained domain can be ineffective. In this paper, we address both single-source and multi-source UDA from a completely different perspective, which is to view each instance as a fine domain. Feature alignment across domains is thus redundant. Instead, we propose to perform dynamic instance domain adaptation (DIDA). Concretely, a dynamic neural network with adaptive convolutional kernels is developed to generate instance-adaptive residuals to adapt domain-agnostic deep features to each individual instance. This enables a shared classifier to be applied to both source and target domain data without relying on any domain annotation. Further, instead of imposing intricate feature alignment losses, we adopt a simple semi-supervised learning paradigm using only a cross-entropy loss for both labeled source and pseudo labeled target data. Our model, dubbed DIDA-Net, achieves state-of-the-art performance on several commonly used single-source and multi-source UDA datasets including Digits, Office-Home, DomainNet, Digit-Five, and PACS.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

FashionViL: Fashion-Focused Vision-and-Language Representation Learning

Large-scale Vision-and-Language (V+L) pre-training for representation learning has proven to be effective in boosting various downstream V+L tasks. However, when it comes to the fashion domain, existing V+L methods are inadequate as they overlook the unique characteristics of both the fashion V+L data and downstream tasks. In this work, we propose a novel fashion-focused V+L representation learning framework, dubbed as FashionViL. It contains two novel fashion-specific pre-training tasks designed particularly to exploit two intrinsic attributes with fashion V+L data. First, in contrast to other domains where a V+L data point contains only a single image-text pair, there could be multiple images in the fashion domain. We thus propose a Multi-View Contrastive Learning task for pulling closer the visual representation of one image to the compositional multimodal representation of another image+text. Second, fashion text (e.g., product description) often contains rich fine-grained concepts (attributes/noun phrases). To exploit this, a Pseudo-Attributes Classification task is introduced to encourage the learned unimodal (visual/textual) representations of the same concept to be adjacent. Further, fashion V+L tasks uniquely include ones that do not conform to the common one-stream or two-stream architectures (e.g., text-guided image retrieval). We thus propose a flexible, versatile V+L model architecture consisting of a modality-agnostic Transformer so that it can be flexibly adapted to any downstream tasks. Extensive experiments show that our FashionViL achieves a new state of the art across five downstream tasks. Code is available at https://github.com/BrandonHanx/mmf.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context

We advance sketch research to scenes with the first dataset of freehand scene sketches, FS-COCO. With practical applications in mind, we collect sketches that convey scene content well but can be sketched within a few minutes by a person with any sketching skills. Our dataset comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals, offering both object- and scene-level abstraction. Each sketch is augmented with its text description. Using our dataset, we study for the first time the problem of fine-grained image retrieval from freehand scene sketches and sketch captions. We draw insights on: (i) Scene salience encoded in sketches using the strokes temporal order; (ii) Performance comparison of image retrieval from a scene sketch and an image caption; (iii) Complementarity of information in sketches and image captions, as well as the potential benefit of combining the two modalities. In addition, we extend a popular vector sketch LSTM-based encoder to handle sketches with larger complexity than was supported by previous work. Namely, we propose a hierarchical sketch decoder, which we leverage at a sketch-specific "pre-text" task. Our dataset enables for the first time research on freehand scene sketch understanding and its practical applications.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Giant and Reversible Electronic Structure Evolution in a Magnetic Topological Material EuCd2As2

The electronic structure and the physical properties of quantum materials can be significantly altered by charge carrier doping and magnetic state transition. Here we report a discovery of a giant and reversible electronic structure evolution with doping in a magnetic topological material. By performing high-resolution angle-resolved photoemission measurements on EuCd2As2,we found that a huge amount of hole doping can be introduced into the sample surface due to surface absorption. The electronic structure exhibits a dramatic change with the hole doping which can not be described by a rigid band shift. Prominent band splitting is observed at high doping which corresponds to a doping-induced magnetic transition at low temperature (below -15 K) from an antiferromagnetic state to a ferromagnetic state. These results have established a detailed electronic phase diagram of EuCd2As2 where the electronic structure and the magnetic structure change systematically and dramatically with the doping level. They further suggest that the transport, magnetic and topological properties of EuCd2As2 can be greatly modified by doping. These work will stimulate further investigations to explore for new phenomena and properties in doping this magnetic topological material.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Magnetic Excitations in Strained Infinite-layer Nickelate PrNiO2

Strongly correlated materials often respond sensitively to the external perturbations. In the recently discovered superconducting infinite-layer nickelates, the superconducting transition temperature can be dramatically enhanced via only ~1% compressive strain-tuning enabled by substrate design. However, the root of such enhancement remains elusive. While the superconducting pairing mechanism is still not settled, magnetic Cooper pairing - similar to the cuprates has been proposed. Using resonant inelastic x-ray scattering, we investigate the magnetic excitations in infinite-layer PrNiO2 thin films for different strain conditions. The magnon bandwidth of PrNiO2 shows only marginal response to strain-tuning, in sharp contrast to the striking enhancement of the superconducting transition temperature Tc in the doped superconducting samples. These results suggest the enhancement of Tc is not mediated by spin excitations and thus provide important empirics for the understanding of superconductivity in infinite-layer nickelates.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Negative Frames Matter in Egocentric Visual Query 2D Localization

The recently released Ego4D dataset and benchmark significantly scales and diversifies the first-person visual perception data. In Ego4D, the Visual Queries 2D Localization task aims to retrieve objects appeared in the past from the recording in the first-person view. This task requires a system to spatially and temporally localize the most recent appearance of a given object query, where query is registered by a single tight visual crop of the object in a different scene. Our study is based on the three-stage baseline introduced in the Episodic Memory benchmark. The baseline solves the problem by detection and tracking: detect the similar objects in all the frames, then run a tracker from the most confident detection result. In the VQ2D challenge, we identified two limitations of the current baseline. (1) The training configuration has redundant computation. Although the training set has millions of instances, most of them are repetitive and the number of unique object is only around 14.6k. The repeated gradient computation of the same object lead to an inefficient training; (2) The false positive rate is high on background frames. This is due to the distribution gap between training and evaluation. During training, the model is only able to see the clean, stable, and labeled frames, but the egocentric videos also have noisy, blurry, or unlabeled background frames. To this end, we developed a more efficient and effective solution. Concretely, we bring the training loop from ~15 days to less than 24 hours, and we achieve 0.17% spatial-temporal AP, which is 31% higher than the baseline. Our solution got the first ranking on the public leaderboard. Our code is publicly available at https://github.com/facebookresearch/vq2d_cvpr.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

One Sketch for All: One-Shot Personalized Sketch Segmentation

We present the first one-shot personalized sketch segmentation method. We aim to segment all sketches belonging to the same category provisioned with a single sketch with a given part annotation while (i) preserving the parts semantics embedded in the exemplar, and (ii) being robust to input style and abstraction. We refer to this scenario as personalized. With that, we importantly enable a much-desired personalization capability for downstream fine-grained sketch analysis tasks. To train a robust segmentation module, we deform the exemplar sketch to each of the available sketches of the same category. Our method generalizes to sketches not observed during training. Our central contribution is a sketch-specific hierarchical deformation network. Given a multi-level sketch-strokes encoding obtained via a graph convolutional network, our method estimates rigid-body transformation from the target to the exemplar, on the upper level. Finer deformation from the exemplar to the globally warped target sketch is further obtained through stroke-wise deformations, on the lower level. Both levels of deformation are guided by mean squared distances between the keypoints learned without supervision, ensuring that the stroke semantics are preserved. We evaluate our method against the state-of-the-art segmentation and perceptual grouping baselines re-purposed for the one-shot setting and against two few-shot 3D shape segmentation methods. We show that our method outperforms all the alternatives by more than $10\%$ on average. Ablation studies further demonstrate that our method is robust to personalization: changes in input part semantics and style differences.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Partially Does It: Towards Scene-Level FG-SBIR with Partial Input

We scrutinise an important observation plaguing scene-level sketch research -- that a significant portion of scene sketches are "partial". A quick pilot study reveals: (i) a scene sketch does not necessarily contain all objects in the corresponding photo, due to the subjective holistic interpretation of scenes, (ii) there exists significant empty (white) regions as a result of object-level abstraction, and as a result, (iii) existing scene-level fine-grained sketch-based image retrieval methods collapse as scene sketches become more partial. To solve this "partial" problem, we advocate for a simple set-based approach using optimal transport (OT) to model cross-modal region associativity in a partially-aware fashion. Importantly, we improve upon OT to further account for holistic partialness by comparing intra-modal adjacency matrices. Our proposed method is not only robust to partial scene-sketches but also yields state-of-the-art performance on existing datasets.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Proposal-Free Temporal Action Detection via Global Segmentation Mask Learning

Existing temporal action detection (TAD) methods rely on generating an overwhelmingly large number of proposals per video. This leads to complex model designs due to proposal generation and/or per-proposal action instance evaluation and the resultant high computational cost. In this work, for the first time, we propose a proposal-free Temporal Action detection model with Global Segmentation mask (TAGS). Our core idea is to learn a global segmentation mask of each action instance jointly at the full video length. The TAGS model differs significantly from the conventional proposal-based methods by focusing on global temporal representation learning to directly detect local start and end points of action instances without proposals. Further, by modeling TAD holistically rather than locally at the individual proposal level, TAGS needs a much simpler model architecture with lower computational cost. Extensive experiments show that despite its simpler design, TAGS outperforms existing TAD methods, achieving new state-of-the-art performance on two benchmarks. Importantly, it is ~ 20x faster to train and ~1.6x more efficient for inference. Our PyTorch implementation of TAGS is available at https://github.com/sauradip/TAGS .

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Quasi-uniaxial pressure induced superconductivity in stoichiometric compound UTe$_2$

The recent discovery of superconductivity in heavy Fermion compound UTe2, a candidate topological and triplet-paired superconductor, has aroused widespread interest. However, to date, there is no consensus on whether the stoichiometric sample of UTe2 is superconducting or not due to lack of reliable evidence to distinguish the difference between the nominal and real compositions of samples. Here, we are the first to clarify that the stoichiometric UT2 is non-superconducting at ambient pressure and under hydrostatic pressure up to 6 GPa, however we find that it can be compressed into superconductivity by application of quasi-uniaxial pressure. Measurements of resistivity, magnetoresistance and susceptibility reveal that the quasi-uniaxial pressure results in a suppression of the Kondo coherent state seen at ambient pressure, and then leads to a superconductivity initially emerged on the ab-plane at 1.5 GPa. At 4.8 GPa, the superconductivity is developed in three crystallographic directions. The superconducting state coexists with an exotic magnetic ordered state that develops just below the onset temperature of the superconducting transition. The discovery of the quasi-uniaxial-pressure-induced superconductivity with exotic magnetic state in the stoichiometric UTe2 not only provide new understandings on this compound, but also highlight the vital role of Te deficiency in developing the superconductivity at ambient pressures.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Semi-Supervised Temporal Action Detection with Proposal-Free Masking

Existing temporal action detection (TAD) methods rely on a large number of training data with segment-level annotations. Collecting and annotating such a training set is thus highly expensive and unscalable. Semi-supervised TAD (SS-TAD) alleviates this problem by leveraging unlabeled videos freely available at scale. However, SS-TAD is also a much more challenging problem than supervised TAD, and consequently much under-studied. Prior SS-TAD methods directly combine an existing proposal-based TAD method and a SSL method. Due to their sequential localization (e.g, proposal generation) and classification design, they are prone to proposal error propagation. To overcome this limitation, in this work we propose a novel Semi-supervised Temporal action detection model based on PropOsal-free Temporal mask (SPOT) with a parallel localization (mask generation) and classification architecture. Such a novel design effectively eliminates the dependence between localization and classification by cutting off the route for error propagation in-between. We further introduce an interaction mechanism between classification and localization for prediction refinement, and a new pretext task for self-supervised model pre-training. Extensive experiments on two standard benchmarks show that our SPOT outperforms state-of-the-art alternatives, often by a large margin. The PyTorch implementation of SPOT is available at https://github.com/sauradip/SPOT

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Sketch3T: Test-Time Training for Zero-Shot SBIR

Zero-shot sketch-based image retrieval typically asks for a trained model to be applied as is to unseen categories. In this paper, we question to argue that this setup by definition is not compatible with the inherent abstract and subjective nature of sketches, i.e., the model might transfer well to new categories, but will not understand sketches existing in different test-time distribution as a result. We thus extend ZS-SBIR asking it to transfer to both categories and sketch distributions. Our key contribution is a test-time training paradigm that can adapt using just one sketch. Since there is no paired photo, we make use of a sketch raster-vector reconstruction module as a self-supervised auxiliary task. To maintain the fidelity of the trained cross-modal joint embedding during test-time update, we design a novel meta-learning based training paradigm to learn a separation between model updates incurred by this auxiliary task from those off the primary objective of discriminative learning. Extensive experiments show our model to outperform state of-the-arts, thanks to the proposed test-time adaption that not only transfers to new categories but also accommodates to new sketching styles.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the "I can't sketch". We consequently design a stroke subset selector that {detects noisy strokes, leaving only those} which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

SOFT: Softmax-free Transformer with Linear Complexity

Vision transformers (ViTs) have pushed the state-of-the-art for various visual recognition tasks by patch-wise image tokenization followed by self-attention. However, the employment of self-attention modules results in a quadratic complexity in both computation and memory usage. Various attempts on approximating the self-attention computation with linear complexity have been made in Natural Language Processing. However, an in-depth analysis in this work shows that they are either theoretically flawed or empirically ineffective for visual recognition. We further identify that their limitations are rooted in keeping the softmax self-attention during approximations. Specifically, conventional self-attention is computed by normalizing the scaled dot-product between token feature vectors. Keeping this softmax operation challenges any subsequent linearization efforts. Based on this insight, for the first time, a softmax-free transformer or SOFT is proposed. To remove softmax in self-attention, Gaussian kernel function is used to replace the dot-product similarity without further normalization. This enables a full self-attention matrix to be approximated via a low-rank matrix decomposition. The robustness of the approximation is achieved by calculating its Moore-Penrose inverse using a Newton-Raphson method. Extensive experiments on ImageNet show that our SOFT significantly improves the computational efficiency of existing ViT variants. Crucially, with a linear complexity, much longer token sequences are permitted in SOFT, resulting in superior trade-off between accuracy and complexity.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Style-Based Global Appearance Flow for Virtual Try-On

Image-based virtual try-on aims to fit an in-shop garment into a clothed person image. To achieve this, a key step is garment warping which spatially aligns the target garment with the corresponding body parts in the person image. Prior methods typically adopt a local appearance flow estimation model. They are thus intrinsically susceptible to difficult body poses/occlusions and large mis-alignments between person and garment images (see Fig.~\ref{fig:fig1}). To overcome this limitation, a novel global appearance flow estimation model is proposed in this work. For the first time, a StyleGAN based architecture is adopted for appearance flow estimation. This enables us to take advantage of a global style vector to encode a whole-image context to cope with the aforementioned challenges. To guide the StyleGAN flow generator to pay more attention to local garment deformation, a flow refinement module is introduced to add local context. Experiment results on a popular virtual try-on benchmark show that our method achieves new state-of-the-art performance. It is particularly effective in a `in-the-wild' application scenario where the reference image is full-body resulting in a large mis-alignment with the garment image (Fig.~\ref{fig:fig1} Top). Code is available at: \url{https://github.com/SenHe/Flow-Style-VTON}.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Towards artificial general intelligence via a multimodal foundation model

The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of human. Despite tremendous success in the AI research, most of existing methods have only single-cognitive ability. To overcome this limitation and take a solid step towards artificial general intelligence (AGI), we develop a foundation model pre-trained with huge multimodal data, which can be quickly adapted for various downstream cognitive tasks. To achieve this goal, we propose to pre-train our foundation model by self-supervised learning with weak semantic correlation data crawled from the Internet and show that promising results can be obtained on a wide range of downstream tasks. Particularly, with the developed model-interpretability tools, we demonstrate that strong imagination ability is now possessed by our foundation model. We believe that our work makes a transformative stride towards AGI, from our common practice of "weak or narrow AI" to that of "strong or generalized AI".

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

UIGR: Unified Interactive Garment Retrieval

Interactive garment retrieval (IGR) aims to retrieve a target garment image based on a reference garment image along with user feedback on what to change on the reference garment. Two IGR tasks have been studied extensively: text-guided garment retrieval (TGR) and visually compatible garment retrieval (VCR). The user feedback for the former indicates what semantic attributes to change with the garment category preserved, while the category is the only thing to be changed explicitly for the latter, with an implicit requirement on style preservation. Despite the similarity between these two tasks and the practical need for an efficient system tackling both, they have never been unified and modeled jointly. In this paper, we propose a Unified Interactive Garment Retrieval (UIGR) framework to unify TGR and VCR. To this end, we first contribute a large-scale benchmark suited for both problems. We further propose a strong baseline architecture to integrate TGR and VCR in one model. Extensive experiments suggest that unifying two tasks in one framework is not only more efficient by requiring a single model only, it also leads to better performance. Code and datasets are available at https://github.com/BrandonHanx/CompFashion.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Zero-Shot Temporal Action Detection via Vision-Language Prompting

Existing temporal action detection (TAD) methods rely on large training data including segment-level annotations, limited to recognizing previously seen classes alone during inference. Collecting and annotating a large training set for each class of interest is costly and hence unscalable. Zero-shot TAD (ZS-TAD) resolves this obstacle by enabling a pre-trained model to recognize any unseen action classes. Meanwhile, ZS-TAD is also much more challenging with significantly less investigation. Inspired by the success of zero-shot image classification aided by vision-language (ViL) models such as CLIP, we aim to tackle the more complex TAD task. An intuitive method is to integrate an off-the-shelf proposal detector with CLIP style classification. However, due to the sequential localization (e.g, proposal generation) and classification design, it is prone to localization error propagation. To overcome this problem, in this paper we propose a novel zero-Shot Temporal Action detection model via Vision-LanguagE prompting (STALE). Such a novel design effectively eliminates the dependence between localization and classification by breaking the route for error propagation in-between. We further introduce an interaction mechanism between classification and localization for improved optimization. Extensive experiments on standard ZS-TAD video benchmarks show that our STALE significantly outperforms state-of-the-art alternatives. Besides, our model also yields superior results on supervised TAD over recent strong competitors. The PyTorch implementation of STALE is available at https://github.com/sauradip/STALE.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning

Most recent few-shot learning (FSL) methods are based on meta-learning with episodic training. In each meta-training episode, a discriminative feature embedding and/or classifier are first constructed from a support set in an inner loop, and then evaluated in an outer loop using a query set for model updating. This query set sample centered learning objective is however intrinsically limited in addressing the lack of training data problem in the support set. In this paper, a novel contrastive prototype learning with augmented embeddings (CPLAE) model is proposed to overcome this limitation. First, data augmentations are introduced to both the support and query sets with each sample now being represented as an augmented embedding (AE) composed of concatenated embeddings of both the original and augmented versions. Second, a novel support set class prototype centered contrastive loss is proposed for contrastive prototype learning (CPL). With a class prototype as an anchor, CPL aims to pull the query samples of the same class closer and those of different classes further away. This support set sample centered loss is highly complementary to the existing query centered loss, fully exploiting the limited training data in each episode. Extensive experiments on several benchmarks demonstrate that our proposed CPLAE achieves new state-of-the-art.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Deep Learning for Person Re-identification: A Survey and Outlook

Person re-identification (Re-ID) aims at retrieving a person of interest across multiple non-overlapping cameras. With the advancement of deep neural networks and increasing demand of intelligent video surveillance, it has gained significantly increased interest in the computer vision community. By dissecting the involved components in developing a person Re-ID system, we categorize it into the closed-world and open-world settings. The widely studied closed-world setting is usually applied under various research-oriented assumptions, and has achieved inspiring success using deep learning techniques on a number of datasets. We first conduct a comprehensive overview with in-depth analysis for closed-world person Re-ID from three different perspectives, including deep feature representation learning, deep metric learning and ranking optimization. With the performance saturation under closed-world setting, the research focus for person Re-ID has recently shifted to the open-world setting, facing more challenging issues. This setting is closer to practical applications under specific scenarios. We summarize the open-world Re-ID in terms of five different aspects. By analyzing the advantages of existing methods, we design a powerful AGW baseline, achieving state-of-the-art or at least comparable performance on twelve datasets for FOUR different Re-ID tasks. Meanwhile, we introduce a new evaluation metric (mINP) for person Re-ID, indicating the cost for finding all the correct matches, which provides an additional criteria to evaluate the Re-ID system for real applications. Finally, some important yet under-investigated open issues are discussed.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Learning to Generate Novel Domains for Domain Generalization

This paper focuses on domain generalization (DG), the task of learning from multiple source domains a model that generalizes well to unseen domains. A main challenge for DG is that the available source domains often exhibit limited diversity, hampering the model's ability to learn to generalize. We therefore employ a data generator to synthesize data from pseudo-novel domains to augment the source domains. This explicitly increases the diversity of available training domains and leads to a more generalizable model. To train the generator, we model the distribution divergence between source and synthesized pseudo-novel domains using optimal transport, and maximize the divergence. To ensure that semantics are preserved in the synthesized data, we further impose cycle-consistency and classification losses on the generator. Our method, L2A-OT (Learning to Augment by Optimal Transport) outperforms current state-of-the-art DG methods on four benchmark datasets.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Local Black-box Adversarial Attacks: A Query Efficient Approach

Adversarial attacks have threatened the application of deep neural networks in security-sensitive scenarios. Most existing black-box attacks fool the target model by interacting with it many times and producing global perturbations. However, global perturbations change the smooth and insignificant background, which not only makes the perturbation more easily be perceived but also increases the query overhead. In this paper, we propose a novel framework to perturb the discriminative areas of clean examples only within limited queries in black-box attacks. Our framework is constructed based on two types of transferability. The first one is the transferability of model interpretations. Based on this property, we identify the discriminative areas of a given clean example easily for local perturbations. The second is the transferability of adversarial examples. It helps us to produce a local pre-perturbation for improving query efficiency. After identifying the discriminative areas and pre-perturbing, we generate the final adversarial examples from the pre-perturbed example by querying the targeted model with two kinds of black-box attack techniques, i.e., gradient estimation and random search. We conduct extensive experiments to show that our framework can significantly improve the query efficiency during black-box perturbing with a high attack success rate. Experimental results show that our attacks outperform state-of-the-art black-box attacks under various system settings.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Magnetic field-tuned quantum criticality in optimally electron-doped cuprate thin films

Antiferromagnetic (AF) spin fluctuations are commonly believed to play a key role in electron pairing of cuprate superconductors. In electron-doped cuprates, it is still in paradox about the interplay among different electronic states in quantum perturbations, especially between superconducting and magnetic states. Here, we report a systematic transport study on cation-optimized La2-xCexCuO4 (x = 0.10) thin films in high magnetic fields. We find an AF quantum phase transition near 60 T, where the Hall number jumps from nH =-x to nH = 1-x, resembling the change of nH at the AF boundary (xAF = 0.14) tuned by Ce doping. In the AF region a spin dependent state manifesting anomalous positive magnetoresistance is observed, which is closely related to superconductivity. Once the AF state is suppressed by magnetic field, a polarized ferromagnetic state is predicted, reminiscent of the recently reported ferromagnetic state at the quantum endpoint of the superconducting dome by Ce doping. The magnetic field that drives phase transitions in a similar but distinct manner to doping thereby provides a unique perspective to understand the quantum criticality of electron-doped cuprates.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Momentum-Resolved Visualization of Electronic Evolution in Doping a Mott Insulator

High temperature superconductivity in cuprates arises from doping a parent Mott insulator by electrons or holes. A central issue is how the Mott gap evolves and the low-energy states emerge with doping. Here we report angle-resolved photoemission spectroscopy measurements on a cuprate parent compound by sequential in situ electron doping. The chemical potential jumps to the bottom of the upper Hubbard band upon a slight electron doping, making it possible to directly visualize the charge transfer band and the full Mott gap region. With increasing doping, the Mott gap rapidly collapses due to the spectral weight transfer from the charge transfer band to the gapped region and the induced low-energy states emerge in a wide energy range inside the Mott gap. These results provide key information on the electronic evolution in doping a Mott insulator and establish a basis for developing microscopic theories for cuprate superconductivity.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Resonating valence bond realization of spin-1 non-Abelian chiral spin liquid on the torus

We propose resonating valence bond wave functions for a spin-1 system on the torus that realize a non-Abelian chiral spin liquid. The wave functions take the form of infinite dimensional matrix product states constructed from conformal blocks of the $\mathrm{SO}(3)_{1}$ Wess-Zumino-Witten model. This means that they are lattice analogues of the bosonic Moore-Read state introduced in fractional quantum Hall systems. The topological order of this system is revealed by explicit construction of three-fold degenerate ground states and analytical computation of the modular S and T matrices.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Universal quantum transition from superconducting to insulating states in pressurized Bi2Sr2CaCu2O8+δ superconductors

Copper oxide superconductors have continually fascinated the communities of condensed matter physics and material sciences because they host the highest ambient-pressure superconducting transition temperature (Tc) and mysterious physics. Searching for the universal correlation between the superconducting state and its normal state or neighboring ground state is believed to be an effective way for finding clues to elucidate the underlying mechanism of the superconductivity. One of the common pictures for the copper oxide superconductors is that a well-behaved metallic phase will present after the superconductivity is entirely suppressed by chemical doping or application of the magnetic field. Here, we report a different observation of universal quantum transition from superconducting state to insulating-like state under pressure in the under-, optimally- and over-doped Bi2212 superconductors with two CuO2 planes in a unit cell. The same phenomenon has been also found in the Bi2201 superconductor with one CuO2 plane and the Bi2223 superconductor with three CuO2 planes in a unit cell. These results not only provide fresh information but also pose a new challenge for achieving a unified understanding on the underlying physics of the high-Tc superconductivity.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

Universal scaling of the critical temperature and the strange-metal scattering rate in unconventional superconductors

Dramatic evolution of properties with minute change in the doping level is a hallmark of the complex chemistry which governs cuprate superconductivity as manifested in the celebrated superconducting domes as well as quantum criticality taking place at precise compositions. The strange metal state, where the resistivity varies linearly with temperature, has emerged as a central feature in the normal state of cuprate superconductors. The ubiquity of this behavior signals an intimate link between the scattering mechanism and superconductivity. However, a clear quantitative picture of the correlation has been lacking. Here, we report observation of quantitative scaling laws between the superconducting transition temperature $T_{\rm c}$ and the scattering rate associated with the strange metal state in electron-doped cuprate $\rm La_{2-x}Ce_xCuO_4$ (LCCO) as a precise function of the doping level. High-resolution characterization of epitaxial composition-spread films, which encompass the entire overdoped range of LCCO has allowed us to systematically map its structural and transport properties with unprecedented accuracy and increment of $Δx = 0.0015$. We have uncovered the relations $T_{\rm c}\sim(x_{\rm c}-x)^{0.5}\sim(A_1^\square)^{0.5}$, where $x_c$ is the critical doping where superconductivity disappears on the overdoped side and $A_1^\square$ is the scattering rate of perfect $T$-linear resistivity per CuO$_2$ plane. We argue that the striking similarity of the $T_{\rm c}$ vs $A_1^\square$ relation among cuprates, iron-based and organic superconductors is an indication of a common mechanism of the strange metal behavior and unconventional superconductivity in these systems.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

AdarGCN: Adaptive Aggregation GCN for Few-Shot Learning

Existing few-shot learning (FSL) methods assume that there exist sufficient training samples from source classes for knowledge transfer to target classes with few training samples. However, this assumption is often invalid, especially when it comes to fine-grained recognition. In this work, we define a new FSL setting termed few-shot fewshot learning (FSFSL), under which both the source and target classes have limited training samples. To overcome the source class data scarcity problem, a natural option is to crawl images from the web with class names as search keywords. However, the crawled images are inevitably corrupted by large amount of noise (irrelevant images) and thus may harm the performance. To address this problem, we propose a graph convolutional network (GCN)-based label denoising (LDN) method to remove the irrelevant images. Further, with the cleaned web images as well as the original clean training images, we propose a GCN-based FSL method. For both the LDN and FSL tasks, a novel adaptive aggregation GCN (AdarGCN) model is proposed, which differs from existing GCN models in that adaptive aggregation is performed based on a multi-head multi-level aggregation module. With AdarGCN, how much and how far information carried by each graph node is propagated in the graph structure can be determined automatically, therefore alleviating the effects of both noisy and outlying training samples. Extensive experiments show the superior performance of our AdarGCN under both the new FSFSL and the conventional FSL settings.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

AFeSe2 (A=Tl, K, Rb, or Cs): Iron-based superconducting analog of the cuprates

It has long been a challenging task to find compounds with similar crystal and electronic structures as cuprate superconductors with low dimensionality and strong antiferromagnetic fluctuations. The parent compounds of cuprate superconductors are Mott insulators with strong in-plane antiferromagnetic exchange interactions between Cu moments. Here we show, based on first-principles density functional calculations, that AFeSe2 (A=Tl, K, Rb, or Cs) exhibit many of the physical properties common to the cuprate parent compounds: (1) the FeSe2 layer in AFeSe2 is similar in crystalline and electronic structures to the CuO2 plane in cuprates, although Se atoms are not coplanar to the square Fe-lattice; (2) they are antiferromagnetic insulators, but with relatively small charge excitation gaps; (3) their ground states are Neel antiferromagnetic ordered, similar as in cuprates; and (4) the antiferromagnetic exchange interactions between Fe moments are larger than in other iron-based superconducting materials, but comparable to those in cuprates. Like cuprates, these compounds may become high-Tc superconductors upon doping of charge carriers either by chemical substitution or intercalation or by liquid or solid gating.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Automatic Differentiation for Second Renormalization of Tensor Networks

Tensor renormalization group (TRG) constitutes an important methodology for accurate simulations of strongly correlated lattice models. Facilitated by the automatic differentiation technique widely used in deep learning, we propose a uniform framework of differentiable TRG ($\partial$TRG) that can be applied to improve various TRG methods, in an automatic fashion. Essentially, $\partial$TRG systematically extends the concept of second renormalization [PRL 103, 160601 (2009)] where the tensor environment is computed recursively in the backward iteration, in the sense that given the forward process of TRG, $\partial$TRG automatically finds the gradient through backpropagation, with which one can deeply "train" the tensor networks. We benchmark $\partial$TRG in solving the square-lattice Ising model, and demonstrate its power by simulating one- and two-dimensional quantum systems at finite temperature. The deep optimization as well as GPU acceleration renders $\partial$TRG manybody simulations with high efficiency and accuracy.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

BézierSketch: A generative model for scalable vector sketches

The study of neural generative models of human sketches is a fascinating contemporary modeling problem due to the links between sketch image generation and the human drawing process. The landmark SketchRNN provided breakthrough by sequentially generating sketches as a sequence of waypoints. However this leads to low-resolution image generation, and failure to model long sketches. In this paper we present BézierSketch, a novel generative model for fully vector sketches that are automatically scalable and high-resolution. To this end, we first introduce a novel inverse graphics approach to stroke embedding that trains an encoder to embed each stroke to its best fit Bézier curve. This enables us to treat sketches as short sequences of paramaterized strokes and thus train a recurrent sketch generator with greater capacity for longer sketches, while producing scalable high-resolution results. We report qualitative and quantitative results on the Quick, Draw! benchmark.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Compressing deep neural networks by matrix product operators

A deep neural network is a parametrization of a multilayer mapping of signals in terms of many alternatively arranged linear and nonlinear transformations. The linear transformations, which are generally used in the fully connected as well as convolutional layers, contain most of the variational parameters that are trained and stored. Compressing a deep neural network to reduce its number of variational parameters but not its prediction power is an important but challenging problem toward the establishment of an optimized scheme in training efficiently these parameters and in lowering the risk of overfitting. Here we show that this problem can be effectively solved by representing linear transformations with matrix product operators (MPOs), which is a tensor network originally proposed in physics to characterize the short-range entanglement in one-dimensional quantum states. We have tested this approach in five typical neural networks, including FC2, LeNet-5, VGG, ResNet, and DenseNet on two widely used data sets, namely, MNIST and CIFAR-10, and found that this MPO representation indeed sets up a faithful and efficient mapping between input and output signals, which can keep or even improve the prediction accuracy with a dramatically reduced number of parameters. Our method greatly simplifies the representations in deep learning, and opens a possible route toward establishing a framework of modern neural networks which might be simpler and cheaper, but more efficient.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Correlation between Fermi surface reconstruction and superconductivity in pressurized FeTe0.55Se0.45

Here we report the first results of the high-pressure Hall coefficient (RH) measurements, combined with the high-pressure resistance measurements, at different temperatures on the putative topological superconductor FeTe0.55Se0.45. We find the intimate correlation of sign change of RH, a fingerprint to manifest the reconstruction of Fermi surface, with structural phase transition and superconductivity. Below the critical pressure (PC) of 2.7 GPa, our data reveal that the hole - electron carriers are thermally balanced (RH=0) at a critical temperature (T*), where RH changes its sign from positive to negative, and concurrently a tetragonal-orthorhombic phase transition takes place. Within the pressure range from 1bar to PC, T* is continuously suppressed by pressure, while TC increases monotonically. At about PC, T* is indistinguishable and TC reaches a maximum value. Moreover, a pressure-induced sign change of RH is found at ~PC where the orthorhombic-monoclinic phase transition occurs. With further compression, TC decreases and disappears at ~ 12 GPa. The correlation among the electron-hole balance, crystal structure and superconductivity found in the pressurized FeTe0.55Se0.45 implies that its nontrivial superconductivity is closely associated with its exotic normal state resulted from the interplay between the reconstruction of the Fermi surface and the change of the structural lattice.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

Sketch as an image search query is an ideal alternative to text in capturing the fine-grained visual details. Prior successes on fine-grained sketch-based image retrieval (FG-SBIR) have demonstrated the importance of tackling the unique traits of sketches as opposed to photos, e.g., temporal vs. static, strokes vs. pixels, and abstract vs. pixel-perfect. In this paper, we study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail -- a person typically sketches up to various extents of detail to depict an object. This hierarchical structure is often visually distinct. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels. In particular, features from a sketch and a photo are enriched using cross-modal co-attention, coupled with hierarchical node fusion at every level to form a better embedding space to conduct retrieval. Experiments on common benchmarks show our method to outperform state-of-the-arts by a significant margin.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Deep Domain-Adversarial Image Generation for Domain Generalisation

Machine learning models typically suffer from the domain shift problem when trained on a source dataset and evaluated on a target dataset of different distribution. To overcome this problem, domain generalisation (DG) methods aim to leverage data from multiple source domains so that a trained model can generalise to unseen domains. In this paper, we propose a novel DG approach based on \emph{Deep Domain-Adversarial Image Generation} (DDAIG). Specifically, DDAIG consists of three components, namely a label classifier, a domain classifier and a domain transformation network (DoTNet). The goal for DoTNet is to map the source training data to unseen domains. This is achieved by having a learning objective formulated to ensure that the generated data can be correctly classified by the label classifier while fooling the domain classifier. By augmenting the source training data with the generated unseen domain data, we can make the label classifier more robust to unknown domain changes. Extensive experiments on four DG datasets demonstrate the effectiveness of our approach.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Domain-Adaptive Few-Shot Learning

Existing few-shot learning (FSL) methods make the implicit assumption that the few target class samples are from the same domain as the source class samples. However, in practice this assumption is often invalid -- the target classes could come from a different domain. This poses an additional challenge of domain adaptation (DA) with few training samples. In this paper, the problem of domain-adaptive few-shot learning (DA-FSL) is tackled, which requires solving FSL and DA in a unified framework. To this end, we propose a novel domain-adversarial prototypical network (DAPN) model. It is designed to address a specific challenge in DA-FSL: the DA objective means that the source and target data distributions need to be aligned, typically through a shared domain-adaptive feature embedding space; but the FSL objective dictates that the target domain per class distribution must be different from that of any source domain class, meaning aligning the distributions across domains may harm the FSL performance. How to achieve global domain distribution alignment whilst maintaining source/target per-class discriminativeness thus becomes the key. Our solution is to explicitly enhance the source/target per-class separation before domain-adaptive feature embedding learning in the DAPN, in order to alleviate the negative effect of domain alignment on FSL. Extensive experiments show that our DAPN outperforms the state-of-the-art FSL and DA models, as well as their naïve combinations. The code is available at https://github.com/dingmyu/DAPN.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Egocentric Action Recognition by Video Attention and Temporal Context

We present the submission of Samsung AI Centre Cambridge to the CVPR2020 EPIC-Kitchens Action Recognition Challenge. In this challenge, action recognition is posed as the problem of simultaneously predicting a single `verb' and `noun' class label given an input trimmed video clip. That is, a `verb' and a `noun' together define a compositional `action' class. The challenging aspects of this real-life action recognition task include small fast moving objects, complex hand-object interactions, and occlusions. At the core of our submission is a recently-proposed spatial-temporal video attention model, called `W3' (`What-Where-When') attention~\cite{perez2020knowing}. We further introduce a simple yet effective contextual learning mechanism to model `action' class scores directly from long-term temporal behaviour based on the `verb' and `noun' prediction scores. Our solution achieves strong performance on the challenge metrics without using object-specific reasoning nor extra training data. In particular, our best solution with multimodal ensemble achieves the 2$^{nd}$ best position for `verb', and 3$^{rd}$ best for `noun' and `action' on the Seen Kitchens test set.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Few-Shot Learning as Domain Adaptation: Algorithm and Analysis

To recognize the unseen classes with only few samples, few-shot learning (FSL) uses prior knowledge learned from the seen classes. A major challenge for FSL is that the distribution of the unseen classes is different from that of those seen, resulting in poor generalization even when a model is meta-trained on the seen classes. This class-difference-caused distribution shift can be considered as a special case of domain shift. In this paper, for the first time, we propose a domain adaptation prototypical network with attention (DAPNA) to explicitly tackle such a domain shift problem in a meta-learning framework. Specifically, armed with a set transformer based attention module, we construct each episode with two sub-episodes without class overlap on the seen classes to simulate the domain shift between the seen and unseen classes. To align the feature distributions of the two sub-episodes with limited training samples, a feature transfer network is employed together with a margin disparity discrepancy (MDD) loss. Importantly, theoretical analysis is provided to give the learning bound of our DAPNA. Extensive experiments show that our DAPNA outperforms the state-of-the-art FSL alternatives, often by significant margins.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Fine-Grained Instance-Level Sketch-Based Video Retrieval

Existing sketch-analysis work studies sketches depicting static objects or scenes. In this work, we propose a novel cross-modal retrieval problem of fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a sketch sequence is used as a query to retrieve a specific target video instance. Compared with sketch-based still image retrieval, and coarse-grained category-level video retrieval, this is more challenging as both visual appearance and motion need to be simultaneously matched at a fine-grained level. We contribute the first FG-SBVR dataset with rich annotations. We then introduce a novel multi-stream multi-modality deep network to perform FG-SBVR under both strong and weakly supervised settings. The key component of the network is a relation module, designed to prevent model over-fitting given scarce training data. We show that this model significantly outperforms a number of existing state-of-the-art models designed for video analysis.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Incremental Few-Shot Object Detection

Most existing object detection methods rely on the availability of abundant labelled training samples per class and offline model training in a batch mode. These requirements substantially limit their scalability to open-ended accommodation of novel classes with limited labelled training data. We present a study aiming to go beyond these limitations by considering the Incremental Few-Shot Detection (iFSD) problem setting, where new classes must be registered incrementally (without revisiting base classes) and with few examples. To this end we propose OpeN-ended Centre nEt (ONCE), a detector designed for incrementally learning to detect novel class objects with few examples. This is achieved by an elegant adaptation of the CentreNet detector to the few-shot learning scenario, and meta-learning a class-specific code generator model for registering novel classes. ONCE fully respects the incremental learning paradigm, with novel class registration requiring only a single forward pass of few-shot training samples, and no access to base classes -- thus making it suitable for deployment on embedded devices. Extensive experiments conducted on both the standard object detection and fashion landmark detection tasks show the feasibility of iFSD for the first time, opening an interesting and very important line of research.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention

Attentive video modeling is essential for action recognition in unconstrained videos due to their rich yet redundant information over space and time. However, introducing attention in a deep neural network for action recognition is challenging for two reasons. First, an effective attention module needs to learn what (objects and their local motion patterns), where (spatially), and when (temporally) to focus on. Second, a video attention module must be efficient because existing action recognition models already suffer from high computational cost. To address both challenges, a novel What-Where-When (W3) video attention module is proposed. Departing from existing alternatives, our W3 module models all three facets of video attention jointly. Crucially, it is extremely efficient by factorizing the high-dimensional video feature data into low-dimensional meaningful spaces (1D channel vector for `what' and 2D spatial tensors for `where'), followed by lightweight temporal attention reasoning. Extensive experiments show that our attention model brings significant improvements to existing action recognition models, achieving new state-of-the-art performance on a number of benchmarks.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

On Learning Semantic Representations for Million-Scale Free-Hand Sketches

In this paper, we study learning semantic representations for million-scale free-hand sketches. This is highly challenging due to the domain-unique traits of sketches, e.g., diverse, sparse, abstract, noisy. We propose a dual-branch CNNRNN network architecture to represent sketches, which simultaneously encodes both the static and temporal patterns of sketch strokes. Based on this architecture, we further explore learning the sketch-oriented semantic representations in two challenging yet practical settings, i.e., hashing retrieval and zero-shot recognition on million-scale sketches. Specifically, we use our dual-branch architecture as a universal representation framework to design two sketch-specific deep models: (i) We propose a deep hashing model for sketch retrieval, where a novel hashing loss is specifically designed to accommodate both the abstract and messy traits of sketches. (ii) We propose a deep embedding model for sketch zero-shot recognition, via collecting a large-scale edge-map dataset and proposing to extract a set of semantic vectors from edge-maps as the semantic knowledge for sketch zero-shot domain alignment. Both deep models are evaluated by comprehensive experiments on million-scale sketches and outperform the state-of-the-art competitors.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least number of strokes possible. We further propose an on-the-fly design that starts retrieving as soon as the user starts drawing. To accomplish this, we devise a reinforcement learning-based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode. Additionally, we introduce a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval. We achieve superior early-retrieval efficiency over state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Tunable giant magnetoresistance in a single-molecule junction

Controlling electronic transport through a single-molecule junction is crucial for molecular electronics or spintronics. In magnetic molecular devices, the spin degree-of-freedom can be used to this end since the magnetic properties of the magnetic ion centers fundamentally impact the transport through the molecules. Here we demonstrate that the electron pathway in a single-molecule device can be selected between two molecular orbitals by varying a magnetic field, giving rise to a tunable anisotropic magnetoresistance up to 93%. The unique tunability of the electron pathways is due to the magnetic reorientation of the transition metal center, resulting in a re-hybridization of molecular orbitals. We obtain the tunneling electron pathways by Kondo effect, which manifests either as a peak or a dip line shape. The energy changes of these spin-reorientations are remarkably low and less than one millielectronvolt. The large tunable anisotropic magnetoresistance could be used to control electronic transport in molecular spintronics.

0 citations0 reviewsNo code linkedOpen access
preprint2019arXiv

Emergent superconductivity in single crystalline $\mathrm{MgTi}_2\mathrm{O}_4$ films via structural engineering

Spinel compounds have demonstrated rich functionalities but rarely shown superconductivity. Here, we report the emergence of superconductivity in the spinel $\mathrm{MgTi}_2\mathrm{O}_4$, known to be an insulator with a complicated order. The superconducting transition is achieved by engineering a superlattice of $\mathrm{MgTi}_2\mathrm{O}_4$ and $\mathrm{SrTiO}_3$. The onset transition temperature in the $\mathrm{MgTi}_2\mathrm{O}_4$ layer can be tuned from 0 to 5 K in such geometry, concurrently with a stretched $c$-axis (from 8.51 to 8.53 Å) compared to the bulk material. Such a positive correlation without saturation suggests ample room for the further enhancement. Intriguingly, the superlattice exhibits isotropic upper critical field $H_{\mathrm{c}2}$ that breaks the Pauli limit, distinct from the highly anisotropic feature of interface superconductivity. The origin of superconductivity in the $\mathrm{MgTi}_2\mathrm{O}_4$ layer is understood in combination with the electron energy loss spectra and the first-principles electronic structure calculations, which point to the birth of superconductivity in the $\mathrm{MgTi}_2\mathrm{O}_4$ layer by preventing the Ti-Ti dimerization. Our discovery not only provides a platform to explore the interplay between the superconductivity and other exotic states, but also opens a new window to realize superconductivity in the spinel compounds as well as other titanium oxides.

0 citations0 reviewsNo code linkedOpen access
preprint2019arXiv

Evidence for an Additional Symmetry Breaking from Direct Observation of Band Splitting in the Nematic State of FeSe Superconductor

The iron-based superconductor FeSe has attracted much recent attention because of its simple crystal structure, distinct electronic structure and rich physics exhibited by itself and its derivatives. Determination of its intrinsic electronic structure is crucial to understand its physical properties and superconductivity mechanism. Both theoretical and experimental studies so far have provided a picture that FeSe consists of one hole-like Fermi surface around the Brillouin zone center in its nematic state. Here we report direct observation of two hole-like Fermi surface sheets around the Brillouin zone center, and the splitting of the associated bands, in the nematic state of FeSe by taking high resolution laser-based angle-resolved photoemission measurements. These results indicate that, in addition to nematic order and spin-orbit coupling, there is an additional order in FeSe that breaks either inversion or time reversal symmetries. The new Fermi surface topology asks for reexamination of the existing theoretical and experimental understanding of FeSe and stimulates further efforts to identify the origin of the hidden order in its nematic state.

0 citations0 reviewsNo code linkedOpen access
preprint2019arXiv

Mott phase in a van der Waals transition-metal halide at single layer limit

Two-dimensional materials offer opportunities for unravelling unprecedented ordered states at single layer limit. Among such ordered states, Mott phase is rarely explored. Here, we report the Mott phase in van der Waals chromium (II) iodide (CrI2) films. High quality CrI2 films with atomically flat surface and macro size are grown on graphitized 6H-SiC(0001) substrate by molecular beam epitaxy. By in situ low temperature scanning tunneling microscopy and spectroscopy (STM/STS), we reveal that the film has a band gap as large as ~3.2 eV, which is nearly thickness independent. Density functional plus dynamic mean field theory calculations suggest that CrI2 films may be a strong Mott insulator with a ferromagnetically ordered ground state. The Mott phase is corroborated by the spectral band splitting, that is consistent with the extended Hubbard model, and gap reduction at charge dopants. Our study provides a platform for studying correlated electron states at single layer limit.

0 citations0 reviewsNo code linkedOpen access
preprint2019arXiv

Non-Volatile Superconductivity in an Insulating Copper Oxide Induced via Ionic Liquid Gating

Manipulating the superconducting states of high-T_c cuprate superconductors in an efficient and reliable way is of great importance for their applications in next-generation electronics. Traditional methods are mostly based on a trial-and-error method that is difficult to implement and time consuming. Here, employing ionic liquid gating, a selective control of volatile and non-volatile superconductivity is achieved in pristine insulating Pr_2CuO_{4\pmδ} film, based on two distinct mechanisms: 1) with positive electric fields, the film can be reversibly switched between non-superconducting and superconducting states, attributed to the carrier doping effect. 2) The film becomes more resistive by applying negative bias voltage up to -4 V, but strikingly, a non-volatile superconductivity is achieved once the gate voltage is removed. Such a persistent superconducting state represents a novel phenomenon in copper oxides, resulting from the doping healing of oxygen vacancies in copper-oxygen planes as unraveled by high-resolution scanning transmission electron microscope and in-situ x-ray diffraction experiments. The effective manipulation and mastering of volatile/non-volatile superconductivity in the same parent cuprate opens the door to more functionalities for superconducting electronics, as well as supplies flexible samples for investigating the nature of quantum phase transitions in high-T_c superconductors.

0 citations0 reviewsNo code linkedOpen access
preprint2019arXiv

Selective Hybridization between Main Band and Superstructure Band in Bi$_2$Sr$_2$CaCu$_2$O$_{8+δ}$ Superconductor

High-resolution laser-based angle-resolved photoemission measurements have been carried out on Bi$_2$Sr$_2$CaCu$_2$O$_{8+δ}$ (Bi2212) and Bi$_2$Sr$_{2-x}$La$_x$CuO$_{6+δ}$ (Bi2201) superconductors. Unexpected hybridization between the main band and the superstructure band in Bi2212 is clearly revealed. In the momentum space where one main Fermi surface intersects with one superstructure Fermi surface, four bands are observed instead of two. The hybridization exists in both superconducting state and normal state, and in Bi2212 samples with different doping levels. Such a hybridization is not observed in Bi2201. This phenomenon can be understood by considering the bilayer splitting in Bi2212, the selective hybridization of two bands with peculiar combinations, and the altered matrix element effects of the hybridized bands. These observations provide strong evidence on the origin of the superstructure band which is intrinsic to the CuO$_2$ planes. Therefore, understanding physical properties and superconductivity mechanism in Bi2212 should consider the complete Fermi surface topology which involves the main bands, the superstructure bands and their interactions.

0 citations0 reviewsNo code linkedOpen access
preprint2019arXiv

Strong coupling superconductivity in trilayer film LiB$_2$C$_2$

Coupling between $σ$-bonding electrons and phonons is generally very strong. To metallize $σ$-electrons provides a promising route to hunt for new high-T$_c$ superconductors. Based on this picture and first-principles density functional calculation with Wannier interpolation for electronic structure and lattice dynamics, we predict that trilayer film LiB$_2$C$_2$ is a good candidate to realize this kind of high-T$_c$ superconductivity. By solving the anisotropic Eliashberg equations, we find that free-standing trilayer LiB$_2$C$_2$ is a phonon-mediated superconductor with T$_c$ exceeding the liquid-nitrogen temperature at ambient pressure. The transition temperature can be further raised to 125 K by applying a biaxial tensile strain.

0 citations0 reviewsNo code linkedOpen access