Source author record

Masayuki Takeda

Explore this author’s publications, collaborators and research topics. This catalog record has not been claimed by its author.

ResearcherUnclaimed source record

Catalog footprint

Research overview

26works
5topics
4close collaborators

Published work

26 publications

preprint2022arXiv

Combinatorics of minimal absent words for a sliding window

A string ww is called a minimal absent word (MAW) for another string TT if ww does not occur in TT but the proper substrings of ww occur in TT. For example, let Σ={a,b,c}Σ= \{\mathtt{a, b, c}\} be the alphabet. Then, the set of MAWs for string w=abaabw = \mathtt{abaab} is {aaa,aaba,bab,bb,c}\{\mathtt{aaa, aaba, bab, bb, c}\}. In this paper, we study combinatorial properties of MAWs in the sliding window model, namely, how the set of MAWs changes when a sliding window of fixed length dd is shifted over the input string TT of length nn, where 1≤d<n1 \leq d < n. We present \emph{tight} upper and lower bounds on the maximum number of changes in the set of MAWs for a sliding window over TT, both in the cases of general alphabets and binary alphabets. Our bounds improve on the previously known best bounds [Crochemore et al., 2020].

preprint2021arXiv

Computing longest palindromic substring after single-character or block-wise edits

Palindromes are important objects in strings which have been extensively studied from combinatorial, algorithmic, and bioinformatics points of views. It is known that the length of the longest palindromic substrings (LPSs) of a given string T of length n can be computed in O(n) time by Manacher's algorithm [J. ACM '75]. In this paper, we consider the problem of finding the LPS after the string is edited. We present an algorithm that uses O(n) time and space for preprocessing, and answers the length of the LPSs in O(\log (\min \{σ, \log n\})) time after a single character substitution, insertion, or deletion, where σdenotes the number of distinct characters appearing in T. We also propose an algorithm that uses O(n) time and space for preprocessing, and answers the length of the LPSs in O(\ell + \log \log n) time, after an existing substring in T is replaced by a string of arbitrary length \ell.

preprint2021arXiv

The Parameterized Suffix Tray

Let ΣΣ and ΠΠ be disjoint alphabets, respectively called the static alphabet and the parameterized alphabet. Two strings xx and yy over Σ∪ΠΣ\cup Π of equal length are said to parameterized match (p-match) if there exists a renaming bijection ff on ΣΣ and ΠΠ which is identity on ΣΣ and maps the characters of xx to those of yy so that the two strings become identical. The indexing version of the problem of finding p-matching occurrences of a given pattern in the text is a well-studied topic in string matching. In this paper, we present a state-of-the-art indexing structure for p-matching called the parameterized suffix tray of an input text TT, denoted by PSTray(T)\mathsf{PSTray}(T). We show that PSTray(T)\mathsf{PSTray}(T) occupies O(n)O(n) space and supports pattern matching queries in O(m+log⁡(σ+π)+occ)O(m + \log (σ+π) + \mathit{occ}) time, where nn is the length of TT, mm is the length of a query pattern PP, ππ is the number of distinct symbols of ∣Π∣|Π| in TT, σσ is the number of distinct symbols of ∣Σ∣|Σ| in TT and occ\mathit{occ} is the number of p-matching occurrences of PP in TT. We also present how to build PSTray(T)\mathsf{PSTray}(T) in O(n)O(n) time from the parameterized suffix tree of TT.

preprint2020arXiv

Detecting kk-(Sub-)Cadences and Equidistant Subsequence Occurrences

The equidistant subsequence pattern matching problem is considered. Given a pattern string PP and a text string TT, we say that PP is an \emph{equidistant subsequence} of TT if PP is a subsequence of the text such that consecutive symbols of PP in the occurrence are equally spaced. We can consider the problem of equidistant subsequences as generalizations of (sub-)cadences. We give bit-parallel algorithms that yield o(n2)o(n^2) time algorithms for finding kk-(sub-)cadences and equidistant subsequences. Furthermore, O(nlog⁡2n)O(n\log^2 n) and O(nlog⁡n)O(n\log n) time algorithms, respectively for equidistant and Abelian equidistant matching for the case ∣P∣=3|P| = 3, are shown. The algorithms make use of a technique that was recently introduced which can efficiently compute convolutions with linear constraints.

preprint2020arXiv

Fast Algorithms for the Shortest Unique Palindromic Substring Problem on Run-Length Encoded Strings

For a string SS, a palindromic substring S[i..j]S[i..j] is said to be a \emph{shortest unique palindromic substring} (SUPS\mathit{SUPS}) for an interval [s,t][s, t] in SS, if S[i..j]S[i..j] occurs exactly once in SS, the interval [i,j][i, j] contains [s,t][s, t], and every palindromic substring containing [s,t][s, t] which is shorter than S[i..j]S[i..j] occurs at least twice in SS. In this paper, we study the problem of answering SUPS\mathit{SUPS} queries on run-length encoded strings. We show how to preprocess a given run-length encoded string RLES\mathit{RLE}_{S} of size mm in O(m)O(m) space and O(mlog⁡σRLES+mlog⁡m/log⁡log⁡m)O(m \log σ_{\mathit{RLE}_{S}} + m \sqrt{\log m / \log\log m}) time so that all SUPSs\mathit{SUPSs} for any subsequent query interval can be answered in O(log⁡m/log⁡log⁡m+α)O(\sqrt{\log m / \log\log m} + α) time, where αα is the number of outputs, and σRLESσ_{\mathit{RLE}_{S}} is the number of distinct runs of RLES\mathit{RLE}_{S}. Additionaly, we consider a variant of the SUPS problem where a query interval is also given in a run-length encoded form. For this variant of the problem, we present two alternative algorithms with faster queries. The first one answers queries in O(log⁡log⁡m/log⁡log⁡log⁡m+α)O(\sqrt{\log\log m /\log\log\log m} + α) time and can be built in O(mlog⁡σRLES+mlog⁡m/log⁡log⁡m)O(m \log σ_{\mathit{RLE}_{S}} + m \sqrt{\log m / \log\log m}) time, and the second one answers queries in O(log⁡log⁡m+α)O(\log \log m + α) time and can be built in O(mlog⁡σRLES)O(m \log σ_{\mathit{RLE}_{S}}) time. Both of these data structures require O(m)O(m) space.

preprint2020arXiv

Faster STR-EC-LCS Computation

The longest common subsequence (LCS) problem is a central problem in stringology that finds the longest common subsequence of given two strings AA and BB. More recently, a set of four constrained LCS problems (called generalized constrained LCS problem) were proposed by Chen and Chao [J. Comb. Optim, 2011]. In this paper, we consider the substring-excluding constrained LCS (STR-EC-LCS) problem. A string ZZ is said to be an STR-EC-LCS of two given strings AA and BB excluding PP if, ZZ is one of the longest common subsequences of AA and BB that does not contain PP as a substring. Wang et al. proposed a dynamic programming solution which computes an STR-EC-LCS in O(mnr)O(mnr) time and space where m=∣A∣,n=∣B∣,r=∣P∣m = |A|, n = |B|, r = |P| [Inf. Process. Lett., 2013]. In this paper, we show a new solution for the STR-EC-LCS problem. Our algorithm computes an STR-EC-LCS in O(n∣Σ∣+(L+1)(m−L+1)r)O(n|Σ| + (L+1)(m-L+1)r) time where ∣Σ∣≤min⁡{m,n}|Σ| \leq \min\{m, n\} denotes the set of distinct characters occurring in both AA and BB, and LL is the length of the STR-EC-LCS. This algorithm is faster than the O(mnr)O(mnr)-time algorithm for short/long STR-EC-LCS (namely, L∈O(1)L \in O(1) or m−L∈O(1)m-L \in O(1)), and is at least as efficient as the O(mnr)O(mnr)-time algorithm for all cases.

preprint2020arXiv

Grammar-compressed Self-index with Lyndon Words

We introduce a new class of straight-line programs (SLPs), named the Lyndon SLP, inspired by the Lyndon trees (Barcelo, 1990). Based on this SLP, we propose a self-index data structure of O(g)O(g) words of space that can be built from a string TT in O(nlg⁡n)O(n \lg n) expected time, retrieving the starting positions of all occurrences of a pattern PP of length mm in O(m+lg⁡mlg⁡n+occlg⁡g)O(m + \lg m \lg n + occ \lg g) time, where nn is the length of TT, gg is the size of the Lyndon SLP for TT, and occocc is the number of occurrences of PP in TT.

preprint2020arXiv

On repetitiveness measures of Thue-Morse words

We show that the size γ(tn)γ(t_n) of the smallest string attractor of the nnth Thue-Morse word tnt_n is 4 for any n≥4n\geq 4, disproving the conjecture by Mantaci et al. [ICTCS 2019] that it is nn. We also show that δ(tn)=103+24−nδ(t_n) = \frac{10}{3+2^{4-n}} for n≥3n \geq 3, where δ(w)δ(w) is the maximum over all k=1,…,∣w∣k = 1,\ldots,|w|, the number of distinct substrings of length kk in ww divided by kk, which is a measure of repetitiveness recently studied by Kociumaka et al. [LATIN 2020]. Furthermore, we show that the number z(tn)z(t_n) of factors in the self-referencing Lempel-Ziv factorization of tnt_n is exactly 2n2n.

preprint2020arXiv

Space-Efficient Algorithms for Computing Minimal/Shortest Unique Substrings

Given a string TT of length nn, a substring u=T[i..j]u = T[i..j] of TT is called a shortest unique substring (SUS) for an interval [s,t][s,t] if (a) uu occurs exactly once in TT, (b) uu contains the interval [s,t][s,t] (i.e. i≤s≤t≤ji \leq s \leq t \leq j), and (c) every substring vv of TT with ∣v∣<∣u∣|v| < |u| containing [s,t][s,t] occurs at least twice in TT. Given a query interval [s,t]⊂[1,n][s, t] \subset [1, n], the interval SUS problem is to output all the SUSs for the interval [s,t][s,t]. In this article, we propose a 4n+o(n)4n + o(n) bits data structure answering an interval SUS query in output-sensitive O(occ)O(\mathit{occ}) time, where occ\mathit{occ} is the number of returned SUSs. Additionally, we focus on the point SUS problem, which is the interval SUS problem for s=ts = t. Here, we propose a ⌈(log⁡23+1)n⌉+o(n)\lceil (\log_2{3} + 1)n \rceil + o(n) bits data structure answering a point SUS query in the same output-sensitive time. We also propose space-efficient algorithms for computing the minimal unique substrings of TT.

preprint2020arXiv

Towards Efficient Interactive Computation of Dynamic Time Warping Distance

The dynamic time warping (DTW) is a widely-used method that allows us to efficiently compare two time series that can vary in speed. Given two strings AA and BB of respective lengths mm and nn, there is a fundamental dynamic programming algorithm that computes the DTW distance for AA and BB together with an optimal alignment in Θ(mn)Θ(mn) time and space. In this paper, we tackle the problem of interactive computation of the DTW distance for dynamic strings, denoted D2TW\mathrm{D^2TW}, where character-wise edit operation (insertion, deletion, substitution) can be performed at an arbitrary position of the strings. Let MM and NN be the sizes of the run-length encoding (RLE) of AA and BB, respectively. We present an algorithm for D2TW\mathrm{D^2TW} that occupies Θ(mN+nM)Θ(mN+nM) space and uses O(m+n+#chg)⊆O(mN+nM)O(m+n+\#_{\mathrm{chg}}) \subseteq O(mN + nM) time to update a compact differential representation DS\mathit{DS} of the DP table per edit operation, where #chg\#_{\mathrm{chg}} denotes the number of cells in DS\mathit{DS} whose values change after the edit operation. Our method is at least as efficient as the algorithm recently proposed by Froese et al. running in Θ(mN+nM)Θ(mN + nM) time, and is faster when #chg\#_{\mathrm{chg}} is smaller than O(mN+nM)O(mN + nM) which, as our preliminary experiments suggest, is likely to be the case in the majority of instances.

preprint2016arXiv

Deterministic sub-linear space LCE data structures with efficient construction

Given a string SS of nn symbols, a longest common extension query LCE(i,j)\mathsf{LCE}(i,j) asks for the length of the longest common prefix of the iith and jjth suffixes of SS. LCE queries have several important applications in string processing, perhaps most notably to suffix sorting. Recently, Bille et al. (J. Discrete Algorithms 25:42-50, 2014, Proc. CPM 2015: 65-76) described several data structures for answering LCE queries that offers a space-time trade-off between data structure size and query time. In particular, for a parameter 1≤τ≤n1 \leq τ\leq n, their best deterministic solution is a data structure of size O(n/τ)O(n/τ) which allows LCE queries to be answered in O(τ)O(τ) time. However, the construction time for all deterministic versions of their data structure is quadratic in nn. In this paper, we propose a deterministic solution that achieves a similar space-time trade-off of O(τmin⁡{log⁡τ,log⁡nτ})O(τ\min\{\logτ,\log\frac{n}τ\}) query time using O(n/τ)O(n/τ) space, but significantly improve the construction time to O(nτ)O(nτ).

preprint2016arXiv

Dynamic index and LZ factorization in compressed space

In this paper, we propose a new \emph{dynamic compressed index} of O(w)O(w) space for a dynamic text TT, where w=O(min⁡(zlog⁡Nlog⁡∗M,N))w = O(\min(z \log N \log^*M, N)) is the size of the signature encoding of TT, zz is the size of the Lempel-Ziv77 (LZ77) factorization of TT, NN is the length of TT, and M≥3NM \geq 3N is an integer that can be handled in constant time under word RAM model. Our index supports searching for a pattern PP in TT in O(∣P∣fA+log⁡wlog⁡∣P∣log⁡∗M(log⁡N+log⁡∣P∣log⁡∗M)+occlog⁡N)O(|P| f_{\mathcal{A}} + \log w \log |P| \log^* M (\log N + \log |P| \log^* M) + \mathit{occ} \log N) time and insertion/deletion of a substring of length yy in O((y+log⁡Nlog⁡∗M)log⁡wlog⁡Nlog⁡∗M)O((y+ \log N\log^* M)\log w \log N \log^* M) time, where fA=O(min⁡{log⁡log⁡Mlog⁡log⁡wlog⁡log⁡log⁡M,log⁡wlog⁡log⁡w})f_{\mathcal{A}} = O(\min \{ \frac{\log\log M \log\log w}{\log\log\log M}, \sqrt{\frac{\log w}{\log\log w}} \}). Also, we propose a new space-efficient LZ77 factorization algorithm for a given text of length NN, which runs in O(NfA+zlog⁡wlog⁡3N(log⁡∗N)2)O(N f_{\mathcal{A}} + z \log w \log^3 N (\log^* N)^2) time with O(w)O(w) working space.

preprint2016arXiv

Dynamic index, LZ factorization, and LCE queries in compressed space

In this paper, we present the following results: (1) We propose a new \emph{dynamic compressed index} of O(w)O(w) space, that supports searching for a pattern PP in the current text in O(∣P∣f(M,w)+log⁡wlog⁡∣P∣log⁡∗M(log⁡N+log⁡∣P∣log⁡∗M)+occlog⁡N)O(|P| f(M,w) + \log w \log |P| \log^* M (\log N + \log |P| \log^* M) + \mathit{occ} \log N) time and insertion/deletion of a substring of length yy in O((y+log⁡Nlog⁡∗M)log⁡wlog⁡Nlog⁡∗M)O((y+ \log N\log^* M)\log w \log N \log^* M) time, where NN is the length of the current text, MM is the maximum length of the dynamic text, zz is the size of the Lempel-Ziv77 (LZ77) factorization of the current text, f(a,b)=O(min⁡{log⁡log⁡alog⁡log⁡blog⁡log⁡log⁡a,log⁡blog⁡log⁡b})f(a,b) = O(\min \{ \frac{\log\log a \log\log b}{\log\log\log a}, \sqrt{\frac{\log b}{\log\log b}} \}) and w=O(zlog⁡Nlog⁡∗M)w = O(z \log N \log^*M). (2) We propose a new space-efficient LZ77 factorization algorithm for a given text of length NN, which runs in O(Nf(N,w′)+zlog⁡w′log⁡3N(log⁡∗N)2)O(N f(N,w') + z \log w' \log^3 N (\log^* N)^2) time with O(w′)O(w') working space, where w′=O(zlog⁡Nlog⁡∗N)w' =O(z \log N \log^* N). (3) We propose a data structure of O(w)O(w) space which supports longest common extension (LCE) queries on the text in O(log⁡N+log⁡ℓlog⁡∗N)O(\log N + \log \ell \log^* N) time, where ℓ\ell is the output LCE length. On top of the above contributions, we show several applications of our data structures which improve previous best known results on grammar-compressed string processing.

preprint2016arXiv

Fully dynamic data structure for LCE queries in compressed space

A Longest Common Extension (LCE) query on a text TT of length NN asks for the length of the longest common prefix of suffixes starting at given two positions. We show that the signature encoding G\mathcal{G} of size w=O(min⁡(zlog⁡Nlog⁡∗M,N))w = O(\min(z \log N \log^* M, N)) [Mehlhorn et al., Algorithmica 17(2):183-198, 1997] of TT, which can be seen as a compressed representation of TT, has a capability to support LCE queries in O(log⁡N+log⁡ℓlog⁡∗M)O(\log N + \log \ell \log^* M) time, where ℓ\ell is the answer to the query, zz is the size of the Lempel-Ziv77 (LZ77) factorization of TT, and M≥4NM \geq 4N is an integer that can be handled in constant time under word RAM model. In compressed space, this is the fastest deterministic LCE data structure in many cases. Moreover, G\mathcal{G} can be enhanced to support efficient update operations: After processing G\mathcal{G} in O(wfA)O(w f_{\mathcal{A}}) time, we can insert/delete any (sub)string of length yy into/from an arbitrary position of TT in O((y+log⁡Nlog⁡∗M)fA)O((y+ \log N\log^* M) f_{\mathcal{A}}) time, where fA=O(min⁡{log⁡log⁡Mlog⁡log⁡wlog⁡log⁡log⁡M,log⁡wlog⁡log⁡w})f_{\mathcal{A}} = O(\min \{ \frac{\log\log M \log\log w}{\log\log\log M}, \sqrt{\frac{\log w}{\log\log w}} \}). This yields the first fully dynamic LCE data structure. We also present efficient construction algorithms from various types of inputs: We can construct G\mathcal{G} in O(NfA)O(N f_{\mathcal{A}}) time from uncompressed string TT; in O(nlog⁡log⁡nlog⁡Nlog⁡∗M)O(n \log\log n \log N \log^* M) time from grammar-compressed string TT represented by a straight-line program of size nn; and in O(zfAlog⁡Nlog⁡∗M)O(z f_{\mathcal{A}} \log N \log^* M) time from LZ77-compressed string TT with zz factors. On top of the above contributions, we show several applications of our data structures which improve previous best known results on grammar-compressed string processing.

preprint2015arXiv

Constructing LZ78 Tries and Position Heaps in Linear Time for Large Alphabets

We present the first worst-case linear-time algorithm to compute the Lempel-Ziv 78 factorization of a given string over an integer alphabet. Our algorithm is based on nearest marked ancestor queries on the suffix tree of the given string. We also show that the same technique can be used to construct the position heap of a set of strings in worst-case linear time, when the set of strings is given as a trie.

preprint2013arXiv

Computing convolution on grammar-compressed text

The convolution between a text string SS of length NN and a pattern string PP of length mm can be computed in O(Nlog⁡m)O(N \log m) time by FFT. It is known that various types of approximate string matching problems are reducible to convolution. In this paper, we assume that the input text string is given in a compressed form, as a \emph{straight-line program (SLP)}, which is a context free grammar in the Chomsky normal form that derives a single string. Given an SLP S\mathcal{S} of size nn describing a text SS of length NN, and an uncompressed pattern PP of length mm, we present a simple O(nmlog⁡m)O(nm \log m)-time algorithm to compute the convolution between SS and PP. We then show that this can be improved to O(min⁡{nm,N−α}log⁡m)O(\min\{nm, N-α\} \log m) time, where α≥0α\geq 0 is a value that represents the amount of redundancy that the SLP captures with respect to the length-mm substrings. The key of the improvement is our new algorithm that computes the convolution between a trie of size rr and a pattern string PP of length mm in O(rlog⁡m)O(r \log m) time.

preprint2013arXiv

Detecting regularities on grammar-compressed strings

We solve the problems of detecting and counting various forms of regularities in a string represented as a Straight Line Program (SLP). Given an SLP of size nn that represents a string ss of length NN, our algorithm compute all runs and squares in ss in O(n3h)O(n^3h) time and O(n2)O(n^2) space, where hh is the height of the derivation tree of the SLP. We also show an algorithm to compute all gapped-palindromes in O(n3h+gnhlog⁡N)O(n^3h + gnh\log N) time and O(n2)O(n^2) space, where gg is the length of the gap. The key technique of the above solution also allows us to compute the periods and covers of the string in O(n2h)O(n^2 h) time and O(nh(n+log⁡2N))O(nh(n+\log^2 N)) time, respectively.

preprint2013arXiv

Efficient Lyndon factorization of grammar compressed text

We present an algorithm for computing the Lyndon factorization of a string that is given in grammar compressed form, namely, a Straight Line Program (SLP). The algorithm runs in O(n4+mn3h)O(n^4 + mn^3h) time and O(n2)O(n^2) space, where mm is the size of the Lyndon factorization, nn is the size of the SLP, and hh is the height of the derivation tree of the SLP. Since the length of the decompressed string can be exponentially large w.r.t. n,mn, m and hh, our result is the first polynomial time solution when the string is given as SLP.

preprint2013arXiv

Faster Compact On-Line Lempel-Ziv Factorization

We present a new on-line algorithm for computing the Lempel-Ziv factorization of a string that runs in O(Nlog⁡N)O(N\log N) time and uses only O(Nlog⁡σ)O(N\logσ) bits of working space, where NN is the length of the string and σσ is the size of the alphabet. This is a notable improvement compared to the performance of previous on-line algorithms using the same order of working space but running in either O(Nlog⁡3N)O(N\log^3N) time (Okanohara & Sadakane 2009) or O(Nlog⁡2N)O(N\log^2N) time (Starikovskaya 2012). The key to our new algorithm is in the utilization of an elegant but less popular index structure called Directed Acyclic Word Graphs, or DAWGs (Blumer et al. 1985). We also present an opportunistic variant of our algorithm, which, given the run length encoding of size mm of a string of length NN, computes the Lempel-Ziv factorization on-line, in O(m⋅min⁡{(log⁡log⁡m)(log⁡log⁡N)log⁡log⁡log⁡N,log⁡mlog⁡log⁡m})O\left(m \cdot \min \left\{\frac{(\log\log m)(\log \log N)}{\log\log\log N}, \sqrt{\frac{\log m}{\log \log m}} \right\}\right) time and O(mlog⁡N)O(m\log N) bits of space, which is faster and more space efficient when the string is run-length compressible.

preprint2013arXiv

Time and Space Efficient Lempel-Ziv Factorization based on Run Length Encoding

We propose a new approach for calculating the Lempel-Ziv factorization of a string, based on run length encoding (RLE). We present a conceptually simple off-line algorithm based on a variant of suffix arrays, as well as an on-line algorithm based on a variant of directed acyclic word graphs (DAWGs). Both algorithms run in O(N+nlog⁡n)O(N+n\log n) time and O(n) extra space, where N is the size of the string, n≤Nn\leq N is the number of RLE factors. The time dependency on N is only in the conversion of the string to RLE, which can be computed very efficiently in O(N) time and O(1) extra space (excluding the output). When the string is compressible via RLE, i.e., n=o(N)n = o(N), our algorithms are, to the best of our knowledge, the first algorithms which require only o(N) extra space while running in o(Nlog⁡N)o(N\log N) time.

preprint2012arXiv

Efficient LZ78 factorization of grammar compressed text

We present an efficient algorithm for computing the LZ78 factorization of a text, where the text is represented as a straight line program (SLP), which is a context free grammar in the Chomsky normal form that generates a single string. Given an SLP of size nn representing a text SS of length NN, our algorithm computes the LZ78 factorization of TT in O(nN+mlog⁡N)O(n\sqrt{N}+m\log N) time and O(nN+m)O(n\sqrt{N}+m) space, where mm is the number of resulting LZ78 factors. We also show how to improve the algorithm so that the nNn\sqrt{N} term in the time and space complexities becomes either nLnL, where LL is the length of the longest LZ78 factor, or (N−α)(N - α) where α≥0α\geq 0 is a quantity which depends on the amount of redundancy that the SLP captures with respect to substrings of SS of a certain length. Since m=O(N/log⁡σN)m = O(N/\log_σN) where σσ is the alphabet size, the latter is asymptotically at least as fast as a linear time algorithm which runs on the uncompressed string when σσ is constant, and can be more efficient when the text is compressible, i.e. when mm and nn are small.

preprint2012arXiv

Speeding-up qq-gram mining on grammar-based compressed texts

We present an efficient algorithm for calculating qq-gram frequencies on strings represented in compressed form, namely, as a straight line program (SLP). Given an SLP T\mathcal{T} of size nn that represents string TT, the algorithm computes the occurrence frequencies of all qq-grams in TT, by reducing the problem to the weighted qq-gram frequencies problem on a trie-like structure of size m=∣T∣−dup(q,T)m = |T|-\mathit{dup}(q,\mathcal{T}), where dup(q,T)\mathit{dup}(q,\mathcal{T}) is a quantity that represents the amount of redundancy that the SLP captures with respect to qq-grams. The reduced problem can be solved in linear time. Since m=O(qn)m = O(qn), the running time of our algorithm is O(min⁡{∣T∣−dup(q,T),qn})O(\min\{|T|-\mathit{dup}(q,\mathcal{T}),qn\}), improving our previous O(qn)O(qn) algorithm when q=Ω(∣T∣/n)q = Ω(|T|/n).

preprint2011arXiv

Computing q-gram Frequencies on Collage Systems

Collage systems are a general framework for representing outputs of various text compression algorithms. We consider the all qq-gram frequency problem on compressed string represented as a collage system, and present an O((q+hlog⁡n)n)O((q+h\log n)n)-time O(qn)O(qn)-space algorithm for calculating the frequencies for all qq-grams that occur in the string. Here, nn and hh are respectively the size and height of the collage system.

preprint2011arXiv

Computing q-gram Non-overlapping Frequencies on SLP Compressed Texts

Length-qq substrings, or qq-grams, can represent important characteristics of text data, and determining the frequencies of all qq-grams contained in the data is an important problem with many applications in the field of data mining and machine learning. In this paper, we consider the problem of calculating the {\em non-overlapping frequencies} of all qq-grams in a text given in compressed form, namely, as a straight line program (SLP). We show that the problem can be solved in O(q2n)O(q^2n) time and O(qn)O(qn) space where nn is the size of the SLP. This generalizes and greatly improves previous work (Inenaga & Bannai, 2009) which solved the problem only for q=2q=2 in O(n4log⁡n)O(n^4\log n) time and O(n3)O(n^3) space.

preprint2011arXiv

Fast qq-gram Mining on SLP Compressed Strings

We present simple and efficient algorithms for calculating qq-gram frequencies on strings represented in compressed form, namely, as a straight line program (SLP). Given an SLP of size nn that represents string TT, we present an O(qn)O(qn) time and space algorithm that computes the occurrence frequencies of qq-grams in TT. Computational experiments show that our algorithm and its variation are practical for small qq, actually running faster on various real string data, compared to algorithms that work on the uncompressed text. We also discuss applications in data mining and classification of string data, for which our algorithms can be useful.

preprint2011arXiv

Restructuring Compressed Texts without Explicit Decompression

We consider the problem of {\em restructuring} compressed texts without explicit decompression. We present algorithms which allow conversions from compressed representations of a string TT produced by any grammar-based compression algorithm, to representations produced by several specific compression algorithms including LZ77, LZ78, run length encoding, and some grammar based compression algorithms. These are the first algorithms that achieve running times polynomial in the size of the compressed input and output representations of TT. Since most of the representations we consider can achieve exponential compression, our algorithms are theoretically faster in the worst case, than any algorithm which first decompresses the string for the conversion.

Research graph

See the researcher in context

Open full explorer

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.