BZPEERReading, trust and collaboration for research
Menu

Discover

GraphFollow connections around a paper, topic or saved view.

Workspaces

HomePick up where your research flow left off.InboxHandle requests, replies and notifications from one place.CollectionsCurate reading lists, evidence sets and shareable dossiers.ResearchSee your private provenance graph, trail and imports.CollaborationMove from discovery into focused discussion rooms.PublishDraft, preview and publish full-text research articles.

Network

ExpertsFind specialists worth following or asking for help.CommunitiesJoin research circles organized around shared work.PartnersSee external organizations active around the same topics.

Opportunities

ListingsBrowse roles, calls and collaborations with clearer fit signals.

Account

Log inReturn to your reading and collaboration workspace.Create accountStart saving work, following people and joining teams.
Log inCreate account

Researcher profile

Neeraj Varshney

Neeraj Varshney contributes to research discovery and scholarly infrastructure.

ResearcherAffiliation not importedOpen to collaborate
Computation and LanguageMachine LearningArtificial Intelligenceeess.SPInformation Theorymath.ITNetworking and Internet ArchitecturePerformance

Trust snapshot

Quick read

Trust 21 - EmergingVerification L1Unclaimed author
14works
0followers
8topics
4close collaborators

Actions

Decide how to stay connected

Follow researcher0

Identity and collaboration

How to connect with this researcher

Claiming links this public author record to a researcher profile and unlocks direct collaboration workflows.

Log in to claim

Direct collaboration

Open a focused conversation when the fit is right

Claim this author entity first to unlock direct invitations.

Research graph

See the researcher in context

Open full explorer

Inspect adjacent work, topics, institutions and collaborators without jumping out to a separate graph page.

Building this graph slice

BZPEER is loading the nearby papers, people, topics and institutions for this page.

Published work

14 published item(s)

preprint2023arXiv

The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness

As Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications, their safety concerns become critical areas of NLP research. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark: a collection of diverse safe and unsafe prompts with carefully designed evaluation methods that facilitate systematic evaluation, comparison, and analysis over 'safety' and 'over-defensiveness.' With SODE, we study a variety of LLM defense strategies over multiple state-of-the-art LLMs, which reveals several interesting and important findings, such as (a) the widely popular 'self-checking' techniques indeed improve the safety against unsafe inputs, but this comes at the cost of extreme over-defensiveness on the safe inputs, (b) providing a safety instruction along with in-context exemplars (of both safe and unsafe inputs) consistently improves safety and also mitigates undue over-defensiveness of the models, (c) providing contextual knowledge easily breaks the safety guardrails and makes the models more vulnerable to generating unsafe responses. Overall, our work reveals numerous such critical findings that we believe will pave the way and facilitate further research in improving the safety of LLMs.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Hybrid Transceiver Design for Tera-Hertz MIMO Systems Relying on Bayesian Learning Aided Sparse Channel Estimation

Hybrid transceiver design in multiple-input multiple-output (MIMO) Tera-Hertz (THz) systems relying on sparse channel state information (CSI) estimation techniques is conceived. To begin with, a practical MIMO channel model is developed for the THz band that incorporates its molecular absorption and reflection losses, as well as its non-line-of-sight (NLoS) rays associated with its diffused components. Subsequently, a novel CSI estimation model is derived by exploiting the angular-sparsity of the THz MIMO channel. Then an orthogonal matching pursuit (OMP)-based framework is conceived, followed by designing a sophisticated Bayesian learning (BL)-based approach for efficient estimation of the sparse THz MIMO channel. The Bayesian Cramer-Rao Lower Bound (BCRLB) is also determined for benchmarking the performance of the CSI estimation techniques developed. Finally, an optimal hybrid transmit precoder and receiver combiner pair is designed, which directly relies on the beamspace domain CSI estimates and only requires limited feedback. Finally, simulation results are provided for quantifying the improved mean square error (MSE), spectral-efficiency (SE) and bit-error rate (BER) performance for transmission on practical THz MIMO channel obtained from the HIgh resolution TRANsmission (HITRAN)-database.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

ILDAE: Instance-Level Difficulty Analysis of Evaluation Data

Knowledge of questions' difficulty level helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions. Can we extract such benefits of instance difficulty in NLP? To this end, we conduct Instance-Level Difficulty Analysis of Evaluation data (ILDAE) in a large-scale setup of 23 datasets and demonstrate its five novel applications: 1) conducting efficient-yet-accurate evaluations with fewer instances saving computational cost and time, 2) improving quality of existing evaluation datasets by repairing erroneous and trivial instances, 3) selecting the best model based on application requirements, 4) analyzing dataset characteristics for guiding future data creation, 5) estimating Out-of-Domain performance reliably. Comprehensive experiments for these applications result in several interesting findings, such as evaluation using just 5% instances (selected via ILDAE) achieves as high as 0.93 Kendall correlation with evaluation using complete dataset and computing weighted accuracy using difficulty scores leads to 5.2% higher correlation with Out-of-Domain performance. We release the difficulty scores and hope our analyses and findings will bring more attention to this important yet understudied field of leveraging instance difficulty in evaluations.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Impact of Multiple Fully-Absorbing Receivers in Molecular Communications

Molecular communication is a promising solution to enable intra-body communications among nanomachines. However, malicious and non-cooperative receivers can degrade the performance, compromising these systems' security. Analyzing the communication and security performance of these systems requires accurate channel models. However, such models are not present in the literature. In this work, we develop an analytical framework to derive the hitting probability of a molecule on a fully absorbing receiver (FAR) in the presence of other FARs, which can be either be cooperative or malicious. We first present an approximate hitting probability expression for the 3-FARs case. A simplified expression is obtained for the case when FARs are symmetrically positioned. Using the derived expressions, we study the impact of malicious receivers on the intended receiver and discuss how to minimize this impact to obtain a secure communication channel. We also study the gain that can be obtained by the cooperation of these FARs. We then present an approach to extend the analysis for a system with N FARs. The derived expressions can be used to analyze and design multiple input/output and secure molecular communication systems.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings

In order to equip NLP systems with selective prediction capability, several task-specific approaches have been proposed. However, which approaches work best across tasks or even if they consistently outperform the simplest baseline 'MaxProb' remains to be explored. To this end, we systematically study 'selective prediction' in a large-scale setup of 17 datasets across several NLP tasks. Through comprehensive experiments under in-domain (IID), out-of-domain (OOD), and adversarial (ADV) settings, we show that despite leveraging additional resources (held-out data/computation), none of the existing approaches consistently and considerably outperforms MaxProb in all three settings. Furthermore, their performance does not translate well across tasks. For instance, Monte-Carlo Dropout outperforms all other approaches on Duplicate Detection datasets but does not fare well on NLI datasets, especially in the OOD setting. Thus, we recommend that future selective prediction approaches should be evaluated across tasks and settings for reliable estimation of their capabilities.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Let the Model Decide its Curriculum for Multitask Learning

Curriculum learning strategies in prior multi-task learning approaches arrange datasets in a difficulty hierarchy either based on human perception or by exhaustively searching the optimal arrangement. However, human perception of difficulty may not always correlate well with machine interpretation leading to poor performance and exhaustive search is computationally expensive. Addressing these concerns, we propose two classes of techniques to arrange training instances into a learning curriculum based on difficulty scores computed via model-based approaches. The two classes i.e Dataset-level and Instance-level differ in granularity of arrangement. Through comprehensive experiments with 12 datasets, we show that instance-level and dataset-level techniques result in strong representations as they lead to an average performance improvement of 4.17% and 3.15% over their respective baselines. Furthermore, we find that most of this improvement comes from correctly answering the difficult instances, implying a greater efficacy of our techniques on difficult tasks.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks

Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Towards Improving Selective Prediction Ability of NLP Systems

It's better to say "I can't answer" than to answer incorrectly. This selective prediction ability is crucial for NLP systems to be reliably deployed in real-world applications. Prior work has shown that existing selective prediction techniques fail to perform well, especially in the out-of-domain setting. In this work, we propose a method that improves probability estimates of models by calibrating them using prediction confidence and difficulty score of instances. Using these two signals, we first annotate held-out instances and then train a calibrator to predict the likelihood of correctness of the model's prediction. We instantiate our method with Natural Language Inference (NLI) and Duplicate Detection (DD) tasks and evaluate it in both In-Domain (IID) and Out-of-Domain (OOD) settings. In (IID, OOD) settings, we show that the representations learned by our calibrator result in an improvement of (15.81%, 5.64%) and (6.19%, 13.9%) over 'MaxProb' -- a selective prediction baseline -- on NLI and DD tasks respectively.

0 citations0 reviewsNo code linkedOpen access
preprint2022arXiv

Unsupervised Natural Language Inference Using PHL Triplet Generation

Transformer-based models achieve impressive performance on numerous Natural Language Inference (NLI) benchmarks when trained on respective training datasets. However, in certain cases, training samples may not be available or collecting them could be time-consuming and resource-intensive. In this work, we address the above challenge and present an explorative study on unsupervised NLI, a paradigm in which no human-annotated training samples are available. We investigate it under three settings: PH, P, and NPH that differ in the extent of unlabeled data available for learning. As a solution, we propose a procedural data generation approach that leverages a set of sentence transformations to collect PHL (Premise, Hypothesis, Label) triplets for training NLI models, bypassing the need for human-annotated training data. Comprehensive experiments with several NLI datasets show that the proposed approach results in accuracies of up to 66.75%, 65.9%, 65.39% in PH, P, and NPH settings respectively, outperforming all existing unsupervised baselines. Furthermore, fine-tuning our model with as little as ~0.1% of the human-annotated training dataset (500 instances) leads to 12.2% higher accuracy than the model trained from scratch on the same 500 instances. Supported by this superior performance, we conclude with a recommendation for collecting high-quality task-specific data.

0 citations0 reviewsNo code linkedOpen access
preprint2021arXiv

On the Performance of the Primary and Secondary Links in a 3-D Underlay Cognitive Molecular Communication

Molecular communication often involves coexisting links where certain links may have priority over others. In this work, we consider a system in three-dimensional (3-D) space with two coexisting communication links, each between a point transmitter and fully-absorbing spherical receiver (FAR), where the one link (termed primary) has priority over the second link (termed secondary). The system implements the underlay cognitive-communication strategy for the co-existence of both links, which use the same type of molecules for information transfer. Mutual influence of FARs existing in the same communication medium results in competition for capturing the information-carrying molecules. In this work, first, we derive an approximate hitting probability equation for a diffusion-limited molecular communication system with two spherical FARs of different sizes considering the effect of molecular degradation. The derived equation is then used for the performance analysis of primary and secondary links in a cognitive molecular communication scenario. We show that the simple transmit control strategy at the secondary transmitter can improve the performance of the overall system. We study the influence of molecular degradation and decision threshold on the system performance. We also show that the systems parameters need to be carefully set to improve the performance of the system.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

3-D Diffusive Molecular Communication with Two Fully-Absorbing Receivers: Hitting Probability and Performance Analysis

Exact analytical channel models for molecular communication via diffusion (MCvD) systems involving multiple fully absorbing receivers (FARs) in a three-dimensional (3- D) medium are hard to obtain due to the mathematical intractability of corresponding diffusion equations. This work, therefore, consider an MCvD system with two spherical FARs in a 3-D diffusion-limited medium and develop several insights using an approximate analytical expression for the hitting probability of information molecule (IM). Further, based on the hitting probability, a novel approximate closed-form analytical expression for the area under the receiver operating characteristic curve (AUC) is derived to analyze the detection performance at each FAR in the presence of other FAR. Finally, simulation results are presented to validate the analytical results using the particle-based and Monte-Carlo simulations and to yield important insights into the MCvD system performance with two FARs

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Performance Analysis of Novel Direct Access Schemes for LEO Satellites Based IoT Network

This paper analyzes the performance of low earth orbit (LEO) satellites based internet-of-things (IoT) network where each IoT node makes use of multiple satellites to communicate with the ground station (GS). In this work, we consider fixed and variable gain amplify-and-forward (AF) relaying protocol at each satellite where the received signal from each IoT node is amplified before transmitting to the terrestrial GS for data processing. To analyze the performance of this novel LEO satellites based direct access architecture, the closed-form expressions for outage probability are derived considering two combining schemes at the GS: (i) selection combining; (ii) maximal ratio combining. Further, to gain more insights for diversity order and coding gain, asymptotic outage probability analysis at high SNR for both schemes is also performed. Finally, simulation results are presented to validate the analytical results derived and also to develop several interesting insights into the system performance.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Simplified Ray Tracing for the Millimeter Wave Channel: A Performance Evaluation

Millimeter-wave (mmWave) communication is one of the cornerstone innovations of fifth-generation (5G) wireless networks, thanks to the massive bandwidth available in these frequency bands. To correctly assess the performance of such systems, however, it is essential to have reliable channel models, based on a deep understanding of the propagation characteristics of the mmWave signal. In this respect, ray tracers can provide high accuracy, at the expense of a significant computational complexity, which limits the scalability of simulations. To address this issue, in this paper we present possible simplifications that can reduce the complexity of ray tracing in the mmWave environment, without significantly affecting the accuracy of the model. We evaluate the effect of such simplifications on link-level metrics, testing different configuration parameters and propagation scenarios.

0 citations0 reviewsNo code linkedOpen access
preprint2020arXiv

Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks

Numerical reasoning is often important to accurately understand the world. Recently, several format-specific datasets have been proposed, such as numerical reasoning in the settings of Natural Language Inference (NLI), Reading Comprehension (RC), and Question Answering (QA). Several format-specific models and architectures in response to those datasets have also been proposed. However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain. In pursuit of this goal, we introduce NUMBERGAME, a multifaceted benchmark to evaluate model performance across numerical reasoning tasks of eight diverse formats. We add four existing question types in our compilation. Two of the new types we add are about questions that require external numerical knowledge, commonsense knowledge and domain knowledge. For building a more practical numerical reasoning system, NUMBERGAME demands four capabilities beyond numerical reasoning: (i) detecting question format directly from data (ii) finding intermediate common format to which every format can be converted (iii) incorporating commonsense knowledge (iv) handling data imbalance across formats. We build several baselines, including a new model based on knowledge hunting using a cheatsheet. However, all baselines perform poorly in contrast to the human baselines, indicating the hardness of our benchmark. Our work takes forward the recent progress in generic system development, demonstrating the scope of these under-explored tasks.

0 citations0 reviewsNo code linkedOpen access