perl -E 'say rindex("bar foo bar foo", "foo")'
^__ found first substring from right side
12
perl -E 'say index("bar foo bar foo", "foo")'
^__ found first substring from left side
4
Answer from mpapec on Stack OverflowThis bug was fixed and will be in 5.22.0 which should be out in May. It didn't make it into the 5.20.x series. That patch is simple and you should be able to apply it to the 5.20.2 or 5.18.2 code base and recompile.
Regexes have the same problem, and it's also been fixed in 5.21.
The work around is to not have 2 gig strings in memory, that's good practice in general. If it's read from a file, perhaps read it in blocks and use index on each block.
If you must have a 2 gig string, use substr() to check it in blocks. Unfortunately this means you have to copy the string in pieces. The code below deliberately initializes $substr outside the loop so Perl doesn't reallocate the memory multiple times.
use v5.18;
use strict;
use warnings;
sub big_index {
my $ref = shift;
my $match = shift;
state $block_size = 2**29;
my $strlen = length($$ref);
my $matchlen = length($match);
# No point in doing extra work if we don't need to.
return index($$ref, $match) if $strlen < $block_size;
my $substr = '';
my $offset = 0;
for(
my $offset = 0;
$offset < $strlen;
$offset += ($block_size - $matchlen - 1)
) {
$substr = substr($$ref, $offset, $block_size);
my $ret = index $substr, $match;
return $ret + $offset if $ret != -1;
}
return -1;
}
Thanks to @Schwern for the basic code. This version avoids copying in the 2 GB string by using substr directly. It supports the additional $position parameter like index. I also did the rindex version.
Call like index/rindex except call with a reference to to source string:
index64(\$string, $match);
index64(\$string, $match, $position);
rindex64(\$string, $match);
rindex64(\$string, $match, $position);
sub index64 {
# Do index on strings > 2GB.
# index in Perl < v5.22 does not work for > 2GB
# Input:
# as index
# Output:
# as index
my $ref = shift;
my $match = shift;
my $pos = shift || 0;
my $block_size = 2**31-1;
my $strlen = length($$ref);
# No point in doing extra work if we don't need to.
if($strlen < $block_size) {
return index($$ref, $match, $pos);
}
my $matchlen = length($match);
my $ret;
my $offset = $pos;
while($offset < $strlen) {
$ret = index(
substr($$ref, $offset, $block_size),
$match, $pos-$offset);
if($ret != -1) {
return $ret + $offset;
}
$offset += ($block_size - $matchlen - 1);
}
return -1;
}
sub rindex64 {
# Do rindex on strings > 2GB.
# rindex in Perl < v5.22 does not work for > 2GB
# Input:
# as rindex
# Output:
# as rindex
my $ref = shift;
my $match = shift;
my $pos = shift;
my $block_size = 2**31-1;
my $strlen = length($$ref);
# Default: search from end
$pos = defined $pos ? $pos : $strlen;
# No point in doing extra work if we don't need to.
if($strlen < $block_size) {
return rindex($$ref, $match, $pos);
}
my $matchlen = length($match);
my $ret;
my $offset = $pos - $block_size + $matchlen;
if($offset < 0) {
# The offset is less than a $block_size
# Set the $offset to 0 and
# Adjust block_size accordingly
$block_size = $block_size + $offset;
$offset = 0;
}
while($offset >= 0) {
$ret = rindex(
substr($$ref, $offset, $block_size),
$match);
if($ret != -1) {
return $ret + $offset;
}
$offset -= ($block_size - $matchlen - 1);
}
return -1;
}