BZPEERPapers, people, evidence and research paths
Menu

Discover

SearchFind papers, people, topics, institutions and opportunities from one place.GraphSee how papers, authors, topics and evidence connect.

Research tools

HomeContinue from the papers, people and topics that matter now.CollectionsTurn saved papers into reading lists, evidence maps and shareable context.ResearchReconstruct the path that led you from search to paper to question.CollaborationMove from a paper, topic or expert request into a focused thread.PublishCreate full-text research articles with metadata, formulas and versions.

Network

ExpertsFind specialists whose work and reviews explain why they can help.CommunitiesJoin research groups organized around papers, topics and shared evidence.

Opportunities

ListingsBrowse roles, calls and collaborations connected to real research context.

Account

Log inReturn to your saved papers, graph views and active threads.Create accountStart saving papers, building a research trail and joining teams.
Log inCreate account

Source author record

Łukasz Dębowski

Łukasz Dębowski appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.

ResearcherUnclaimed source record
Information Theorymath.ITComputation and Languagecond-mat.stat-mechmath.PRphysics.data-an

Catalog footprint

What is connected

13works
6topics
2close collaborators

Actions

Connect this record

Log in to claim
Open graphBrowse works

Research graph

See the researcher in context

Open full explorer

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

Published work

13 published item(s)

preprint2020arXiv

On a Class of Markov Order Estimators Based on PPM and Other Universal Codes

We investigate a class of estimators of the Markov order for stationary ergodic processes which form a slight modification of the constructions by Merhav, Gutman, and Ziv in 1989 as well as by Ryabko, Astola, and Malyutov in 2006 and 2016. All the considered estimators compare the estimate of the entropy rate given by a universal code with the empirical conditional entropy of a string and return the order for which the two quantities are approximately equal. However, our modification, which we call universal Markov orders, satisfies a few attractive properties, not shown by the mentioned authors for their original constructions. Firstly, the universal Markov orders are almost surely consistent, without any restrictions. Secondly, they are upper bounded asymptotically by the logarithm of the string length divided by the entropy rate. Thirdly, if we choose the Prediction by Partial Matching (PPM) as the universal code then the number of distinct substrings of the length equal to the universal Markov order constitutes an upper bound for the block mutual information. Thus universal Markov orders can be also used indirectly for quantification of long memory for an ergodic process.

0 citations0 reviewsNo code linkedOpen access
preprint2017arXiv

Is Natural Language a Perigraphic Process? The Theorem about Facts and Words Revisited

As we discuss, a stationary stochastic process is nonergodic when a random persistent topic can be detected in the infinite random text sampled from the process, whereas we call the process strongly nonergodic when an infinite sequence of independent random bits, called probabilistic facts, is needed to describe this topic completely. Replacing probabilistic facts with an algorithmically random sequence of bits, called algorithmic facts, we adapt this property back to ergodic processes. Subsequently, we call a process perigraphic if the number of algorithmic facts which can be inferred from a finite text sampled from the process grows like a power of the text length. We present a simple example of such a process. Moreover, we demonstrate an assertion which we call the theorem about facts and words. This proposition states that the number of probabilistic or algorithmic facts which can be inferred from a text drawn from a process must be roughly smaller than the number of distinct word-like strings detected in this text by means of the PPM compression algorithm. We also observe that the number of the word-like strings for a sample of plays by Shakespeare follows an empirical stepwise power law, in a stark contrast to Markov processes. Hence we suppose that natural language considered as a process is not only non-Markov but also perigraphic.

0 citations0 reviewsNo code linkedOpen access
preprint2017arXiv

Maximal Repetition and Zero Entropy Rate

Maximal repetition of a string is the maximal length of a repeated substring. This paper investigates maximal repetition of strings drawn from stochastic processes. Strengthening previous results, two new bounds for the almost sure growth rate of maximal repetition are identified: an upper bound in terms of conditional Rényi entropy of order $γ>1$ given a sufficiently long past and a lower bound in terms of unconditional Shannon entropy ($γ=1$). Both the upper and the lower bound can be proved using an inequality for the distribution of recurrence time. We also supply an alternative proof of the lower bound which makes use of an inequality for the expectation of subword complexity. In particular, it is shown that a power-law logarithmic growth of maximal repetition with respect to the string length, recently observed for texts in natural language, may hold only if the conditional Rényi entropy rate given a sufficiently long past equals zero. According to this observation, natural language cannot be faithfully modeled by a typical hidden Markov process, which is a class of basic language models used in computational linguistics.

0 citations0 reviewsNo code linkedOpen access
preprint2016arXiv

Consistency of the Plug-In Estimator of the Entropy Rate for Ergodic Processes

A plug-in estimator of entropy is the entropy of the distribution where probabilities of symbols or blocks have been replaced with their relative frequencies in the sample. Consistency and asymptotic unbiasedness of the plug-in estimator can be easily demonstrated in the IID case. In this paper, we ask whether the plug-in estimator can be used for consistent estimation of the entropy rate $h$ of a stationary ergodic process. The answer is positive if, to estimate block entropy of order $k$, we use a sample longer than $2^{k(h+ε)}$, whereas it is negative if we use a sample shorter than $2^{k(h-ε)}$. In particular, if we do not know the entropy rate $h$, it is sufficient to use a sample of length $(|X|+ε)^{k}$ where $|X|$ is the alphabet size. The result is derived using $k$-block coding. As a by-product of our technique, we also show that the block entropy of a stationary process is bounded above by a nonlinear function of the average block entropy of its ergodic components. This inequality can be used for an alternative proof of the known fact that the entropy rate a stationary process equals the average entropy rate of its ergodic components.

0 citations0 reviewsNo code linkedOpen access
preprint2016arXiv

Regular Hilberg Processes: An Example of Processes with a Vanishing Entropy Rate

A regular Hilberg process is a stationary process that satisfies both a hyperlogarithmic growth of maximal repetition and a power-law growth of topological entropy, which are a kind of dual conditions. The hyperlogarithmic growth of maximal repetition has been experimentally observed for texts in natural language, whereas the power-law growth of topological entropy implies a vanishing Shannon entropy rate and thus probably does not hold for natural language. In this paper, we provide a constructive example of regular Hilberg processes, which we call random hierarchical association (RHA) processes. Our construction does not apply the standard cutting and stacking method. For the constructed RHA processes, we demonstrate that the expected length of any uniquely decodable code is orders of magnitude larger than the Shannon block entropy of the ergodic component of the RHA process. Our proposition supplements the classical result by Shields concerning nonexistence of universal redundancy rates.

0 citations0 reviewsNo code linkedOpen access
preprint2015arXiv

A Preadapted Universal Switch Distribution for Testing Hilberg's Conjecture

Hilberg's conjecture about natural language states that the mutual information between two adjacent long blocks of text grows like a power of the block length. The exponent in this statement can be upper bounded using the pointwise mutual information estimate computed for a carefully chosen code. The bound is the better, the lower the compression rate is but there is a requirement that the code be universal. So as to improve a received upper bound for Hilberg's exponent, in this paper, we introduce two novel universal codes, called the plain switch distribution and the preadapted switch distribution. Generally speaking, switch distributions are certain mixtures of adaptive Markov chains of varying orders with some additional communication to avoid so called catch-up phenomenon. The advantage of these distributions is that they both achieve a low compression rate and are guaranteed to be universal. Using the switch distributions we obtain that a sample of a text in English is non-Markovian with Hilberg's exponent being $\le 0.83$, which improves over the previous bound $\le 0.94$ obtained using the Lempel-Ziv code.

0 citations0 reviewsNo code linkedOpen access
preprint2013arXiv

Constant conditional entropy and related hypotheses

Constant entropy rate (conditional entropies must remain constant as the sequence length increases) and uniform information density (conditional probabilities must remain constant as the sequence length increases) are two information theoretic principles that are argued to underlie a wide range of linguistic phenomena. Here we revise the predictions of these principles to the light of Hilberg's law on the scaling of conditional entropy in language and related laws. We show that constant entropy rate (CER) and two interpretations for uniform information density (UID), full UID and strong UID, are inconsistent with these laws. Strong UID implies CER but the reverse is not true. Full UID, a particular case of UID, leads to costly uncorrelated sequences that are totally unrealistic. We conclude that CER and its particular cases are incomplete hypotheses about the scaling of conditional entropies.

0 citations0 reviewsNo code linkedOpen access
preprint2012arXiv

On Hidden Markov Processes with Infinite Excess Entropy

We investigate stationary hidden Markov processes for which mutual information between the past and the future is infinite. It is assumed that the number of observable states is finite and the number of hidden states is countably infinite. Under this assumption, we show that the block mutual information of a hidden Markov process is upper bounded by a power law determined by the tail index of the hidden state distribution. Moreover, we exhibit three examples of processes. The first example, considered previously, is nonergodic and the mutual information between the blocks is bounded by the logarithm of the block length. The second example is also nonergodic but the mutual information between the blocks obeys a power law. The third example obeys the power law and is ergodic.

0 citations0 reviewsNo code linkedOpen access
preprint2011arXiv

Excess entropy in natural language: present state and perspectives

We review recent progress in understanding the meaning of mutual information in natural language. Let us define words in a text as strings that occur sufficiently often. In a few previous papers, we have shown that a power-law distribution for so defined words (a.k.a. Herdan's law) is obeyed if there is a similar power-law growth of (algorithmic) mutual information between adjacent portions of texts of increasing length. Moreover, the power-law growth of information holds if texts describe a complicated infinite (algorithmically) random object in a highly repetitive way, according to an analogous power-law distribution. The described object may be immutable (like a mathematical or physical constant) or may evolve slowly in time (like cultural heritage). Here we reflect on the respective mathematical results in a less technical way. We also discuss feasibility of deciding to what extent these results apply to the actual human communication.

0 citations0 reviewsNo code linkedOpen access
preprint2011arXiv

Mixing, Ergodic, and Nonergodic Processes with Rapidly Growing Information between Blocks

We construct mixing processes over an infinite alphabet and ergodic processes over a finite alphabet for which Shannon mutual information between adjacent blocks of length $n$ grows as $n^β$, where $β\in(0,1)$. The processes are a modification of nonergodic Santa Fe processes, which were introduced in the context of natural language modeling. The rates of mutual information for the latter processes are alike and also established in this paper. As an auxiliary result, it is shown that infinite direct products of mixing processes are also mixing.

0 citations0 reviewsNo code linkedOpen access
preprint2011arXiv

On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts

The article presents a new interpretation for Zipf-Mandelbrot's law in natural language which rests on two areas of information theory. Firstly, we construct a new class of grammar-based codes and, secondly, we investigate properties of strongly nonergodic stationary processes. The motivation for the joint discussion is to prove a proposition with a simple informal statement: If a text of length $n$ describes $n^β$ independent facts in a repetitive way then the text contains at least $n^β/\log n$ different words, under suitable conditions on $n$. In the formal statement, two modeling postulates are adopted. Firstly, the words are understood as nonterminal symbols of the shortest grammar-based encoding of the text. Secondly, the text is assumed to be emitted by a finite-energy strongly nonergodic source whereas the facts are binary IID variables predictable in a shift-invariant way.

0 citations0 reviewsNo code linkedOpen access
preprint2011arXiv

Variable-Length Coding of Two-Sided Asymptotically Mean Stationary Measures

We collect several observations that concern variable-length coding of two-sided infinite sequences in a probabilistic setting. Attention is paid to images and preimages of asymptotically mean stationary measures defined on subsets of these sequences. We point out sufficient conditions under which the variable-length coding and its inverse preserve asymptotic mean stationarity. Moreover, conditions for preservation of shift-invariant $σ$-fields and the finite-energy property are discussed and the block entropies for stationary means of coded processes are related in some cases. Subsequently, we apply certain of these results to construct a stationary nonergodic process with a desired linguistic interpretation.

0 citations0 reviewsNo code linkedOpen access
preprint2009arXiv

Valence extraction using EM selection and co-occurrence matrices

This paper discusses two new procedures for extracting verb valences from raw texts, with an application to the Polish language. The first novel technique, the EM selection algorithm, performs unsupervised disambiguation of valence frame forests, obtained by applying a non-probabilistic deep grammar parser and some post-processing to the text. The second new idea concerns filtering of incorrect frames detected in the parsed text and is motivated by an observation that verbs which take similar arguments tend to have similar frames. This phenomenon is described in terms of newly introduced co-occurrence matrices. Using co-occurrence matrices, we split filtering into two steps. The list of valid arguments is first determined for each verb, whereas the pattern according to which the arguments are combined into frames is computed in the following stage. Our best extracted dictionary reaches an $F$-score of 45%, compared to an $F$-score of 39% for the standard frame-based BHT filtering.

0 citations0 reviewsNo code linkedOpen access