We maintain a website that uses the letters æ, ø, and å in some page addresses. And it worked perfectly, except for some IE questions early on, so far. The problem we got over the last couple of weeks is that search engines, especially Bing, seem to encode letters over and over again.
So we get 404 errors,because the crawler is trying to access the address / butikk / m% C3% 83% C6% 92% C3% 86% E2% 80% 99% C3% 83% E2% 80% A0% C3% A2% E2% 82% AC % E2% 84% A2% C3% 83% C6% 92% C3% A2% E2% 82% AC% C5% A1% C3% 83% E2% 80% 9A% C3% 82% C2% A3% C3% 83 % C6% 92% C3% 86% E2% 80% 99% C3% 83% C2% A2% C3% A2% E2% 80% 9A% C2% AC% C3% 82% C2% A0% C3% 83% C6 % 92% C3% 82% C2% A2% C3% 83% C2% A2% C3% A2% E2% 82% AC% C5% A1% C3% 82% C2% AC% C3% 83% C2% A2% C3 % A2% E2% 82% AC% C5% BE% C3% 82% C2% A2% C3% 83% C6% 92% C3% 86% E2% 80% 99% C3% 83% E2% 80% A0% C3 % A2% E2% 82% AC% E2% 84% A2% C3% 83% C6% 92% C3% A2% E2% 82% AC% C5% A1% C3% 83% E2% 80% 9A% C3% 82 % C2% A2% C3% 83% C6% 92% C3% 86% E2% 80% 99% C3% 83% C2% A2% C3% A2% E2% 80% 9A% C2% AC% C3% 85% C2 % A1% C3% 83% C6% 92% C3% A2% E2% 82% AC% C5% A1% C3% 83% E2% 80% 9A% C3% 82% C2% B8bler, instead of / butikk / m øbler.Using / butikk / m% c3% b8bler would also lead you to the correct page. And since we use the Play Framework, we also get a website error, since our controllers can be no more than 250 characters, but this is not a problem.
Initially, the site did not have a site. We added one, with UTF-8 encoded addresses, hoping that this would lead to the correct bot path, but so far nothing.
So, did someone have a similar problem and solve it, or have some suggestions on what we can do to make Bing Bot use the correct addresses? Any help would be appreciated.
Information added:
Looking at Bing Webmaster Tools, I see that Bing indexed the right address and version with "ø" instead of "ø". Thus, my problem, hopefully, will be resolved by removing the erroneous address from the index.
source
share