Jump to content
Mudsaf

preg_replace issue, unicode block?

Recommended Posts

So i'm not still really familiar with regex and how does it work, but i have issue with my code. I have ¤ in my text and it gets replace with unicode ? block. https://en.wikipedia.org/wiki/Specials_(Unicode_block) for me and I don't get why.

Here is the code I'm using.

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<style>
#kool { width:800px; border:groove; margin:0 auto; padding:25px; }
</style>
</head>
<body>
<?php
/* Test preg_replace function */
$preg_pattern = "/[^A-Za-z0-9!\"#%£&()=@\s]/";

if (isset($_GET['preg'])) {
echo "<div id='kool'>";
echo "<b>Original string: </b> " . $_GET['preg'];
echo "<hr>";
echo "<b>Preg pattern: </b> " . $preg_pattern;
echo "<hr>";
echo "<b>Result: </b> " . preg_replace($preg_pattern,"",$_GET['preg']);
echo "</div>";
}

?>
<form method="get" action="preg_test.php">
<input type="text" name="preg" <?php if (isset($_GET['preg'])) { echo " value='" . $_GET['preg'] . "'"; }  ?>> <input type="submit">
</form>
</body>
</html>

And here is the string I'm trying to use.

Test 0-9, Specialcharacters: !"#¤%&/()=?

Result image below

result.thumb.PNG.29437c96300a4d2bb92a51bfdb495ebe.PNG

Any guides / help appreciated.

Share this post


Link to post
Share on other sites

Your code editor most likely has not encoded the file as UTF-8. Depending on which code editor you're using the way to do this is different. On Windows Notepad, there's an "encoding" dropdown in the same dialog which you should set to "UTF-8". Other text editors have encoding menus or an encoding option in the document properties.

Share this post


Link to post
Share on other sites

Changed to UTF-8 (not sure if it were already), issue still persists somehow. The odd part is, this works when the £ is not added in the preg pattern.

result3.PNG.7c6868eddfb179e2a2dbf2f8401b6f69.PNG

 

Edited by Mudsaf
had wrong image

Share this post


Link to post
Share on other sites

Apparently my browser forces ¤ to be at url, even though i replace it with %C2%A4 (if this was what you meant). Rest of the stuff is encoded properly (chrome). But on edge it is encoded.

result4.thumb.PNG.d60fbabbfe5caddcaa3dacb1cc4233cd.PNG

(Image from Edge browser)

Also  tried urldecode the GET parameter.

Edited by Mudsaf

Share this post


Link to post
Share on other sites

How would that work with preg_replace()?

Share this post


Link to post
Share on other sites

Probably not, but since

"URLs can only be sent over the Internet using the ASCII character-set.

Since URLs often contain characters outside the ASCII set, the URL has to be converted into a valid ASCII format."

You ideally should be filtering a url that gives you the same result, not sometimes one or the other.

Share this post


Link to post
Share on other sites
On 8/19/2019 at 8:57 PM, justsomeguy said:

I would use ord or mb_ord to loop through the characters in the string and print the value of each byte to see what's actually there.

https://www.php.net/manual/en/function.ord.php

https://www.php.net/manual/en/function.mb-ord.php

The unicode block symbol returned value of 194 via ord() function, got any idea what might be the cause to create that unicode block?

result5.PNG.70509b379bf449f6caaa4731ba09c7a5.PNG

Source code of tester below, string too.

 

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<style>
#kool { width:800px; border:groove; margin:0 auto; padding:25px; }
.mid { margin:0 auto; max-width:800px; text-align:centeR; margin-top:10px; }
#preg_input { width:300px; }
</style>
</head>
<body>
<?php
/* Test preg_replace function */

// #%&()=@£\$€\[\]_\-,.:?
$basic = "A-Za-Z0-9";
$preg_pattern = "/[^A-Za-z0-9!\"#%£&()=@\s]/";
$func_preg_replace = preg_replace($preg_pattern,"",$_GET['preg']);
if (isset($_GET['preg'])) {
echo "<div id='kool'>";
echo "<b>Original string: </b> " . $_GET['preg'];
echo "<hr>";
echo "<b>Preg pattern: </b> " . $preg_pattern;
echo "<hr>";
echo "<b>Result: </b> " . $func_preg_replace;
echo "<hr>";
echo "<b>Rawurl: </b> " . rawurlencode($_GET['preg']);
echo "<hr>";
echo "<b>Ord: </b> ";
	for ($i=0;$i<strlen($func_preg_replace);$i++) {
	echo $func_preg_replace[$i] . "(" . ord($func_preg_replace[$i]) .  ") ";	
	}
echo "</div>";
}

?>
<div class="mid">
<form method="get" action="preg_test.php">
<input type="text" id="preg_input" name="preg" <?php if (isset($_GET['preg'])) { echo " value='" . $_GET['preg'] . "'"; }  ?>> <input type="submit">
</form>
</div>
</body>
</html>

And the string without

What is love 0-9, Specialcharacters: !"#¤%&/()=?

Does it reproduce for you guys?

----

Also tried to UTF-8 encode string via php, the preg_replace string, 2 new unicode blocks appeared with �(195) �(130).

$func_preg_replace = utf8_encode ($func_preg_replace);

 

Edited by Mudsaf

Share this post


Link to post
Share on other sites

What about the original string?  Is the 194 part of a code point that only gets partially replaced?

Share this post


Link to post
Share on other sites

Original string returns 2 unicode blocks (straight from $_GET): �(194) �(164). Where ¤ is.

Share this post


Link to post
Share on other sites

Add the u modifier to your pattern:

/[^A-Za-z0-9!\"#%£&()=@\s]/u

  • Like 1

Share this post


Link to post
Share on other sites
7 minutes ago, justsomeguy said:

Add the u modifier to your pattern:

 


/[^A-Za-z0-9!\"#%£&()=@\s]/u

 

Not sure what kind of sorcery is this, but it works now. Thank you!

Share this post


Link to post
Share on other sites

From the manual:

u (PCRE_UTF8)
This modifier turns on additional functionality of PCRE that is incompatible with Perl. Pattern and subject strings are treated as UTF-8. An invalid subject will cause the preg_* function to match nothing; an invalid pattern will trigger an error of level E_WARNING. Five and six octet UTF-8 sequences are regarded as invalid since PHP 5.3.4 (resp. PCRE 7.3 2007-08-28); formerly those have been regarded as valid UTF-8.
  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...