path: root/doc/TODO.detail/locale

From pgsql-patches-owner+M16987@postgresql.org Thu Aug  4 17:35:52 2005
Return-path: <pgsql-patches-owner+M16987@postgresql.org>
Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j74LZot06577
	for <pgman@candle.pha.pa.us>; Thu, 4 Aug 2005 17:35:50 -0400 (EDT)
Received: from localhost (unknown [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id ED433529DB;
	Thu,  4 Aug 2005 18:35:44 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 70225-08; Thu,  4 Aug 2005 21:35:44 +0000 (GMT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by svr1.postgresql.org (Postfix) with ESMTP id 0ECB452998;
	Thu,  4 Aug 2005 18:35:44 -0300 (ADT)
X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org
Received: from localhost (unknown [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id AFAF752840
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>; Thu,  4 Aug 2005 18:33:25 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 70920-01
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>;
	Thu,  4 Aug 2005 21:33:13 +0000 (GMT)
Received: from mx-2.sollentuna.net (mx-2.sollentuna.net [195.84.163.199])
	by svr1.postgresql.org (Postfix) with ESMTP id E1C935280D
	for <pgsql-patches@postgresql.org>; Thu,  4 Aug 2005 18:33:11 -0300 (ADT)
Received: from ALGOL.sollentuna.se (janus.sollentuna.se [62.65.68.67])
	by mx-2.sollentuna.net (Postfix) with ESMTP id 7F5A08F289
	for <pgsql-patches@postgresql.org>; Thu,  4 Aug 2005 23:33:13 +0200 (CEST)
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01C5993C.1E1CB100"
Subject: [PATCHES] FW: Win32 unicode vs ICU
X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0
Date: Thu, 4 Aug 2005 23:33:12 +0200
Message-ID: <6BCB9D8A16AC4241919521715F4D8BCE094656@algol.sollentuna.se>
X-MS-Has-Attach: yes
Thread-Topic: Win32 unicode vs ICU
Thread-Index: AcWVyKf6M9GrCqZAQGeSvCZNg9n/jgDc1WwQ
From: "Magnus Hagander" <mha@sollentuna.net>
To: "PostgreSQL-patches" <pgsql-patches@postgresql.org>
X-Virus-Scanned: by amavisd-new at hub.org
X-Mailing-List: pgsql-patches
List-Archive: <http://archives.postgresql.org/pgsql-patches>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-patches.postgresql.org>
List-Owner: <mailto:pgsql-patches-owner@postgresql.org>
List-Post: <mailto:pgsql-patches@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-patches>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-patches>
Precedence: bulk
Sender: pgsql-patches-owner@postgresql.org
X-Virus-Scanned: by amavisd-new at hub.org
Status: ORr

This is a multi-part message in MIME format.

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I just realised this mail didn't go through. Probably because it was too
large for -hackers. So: repost to -patches. Sorry about that. If it's a
duplicate, even more sorry, but I couldn't find it in the archives.

(This may explain that nobody answered me :P)

//Magnus
=20

> -----Original Message-----
> From: Magnus Hagander=20
> Sent: Sunday, July 31, 2005 2:09 PM
> To: PostgreSQL-development
> Cc: pgsql-hackers-win32@postgresql.org
> Subject: Win32 unicode vs ICU
>=20
> Hi!
>=20
> I've been working with Palles ICU patch to make it work on=20
> win32, and I believe I have it done. While doing it I noticed=20
> that ICU basically converts to UTF16 and back - I previously=20
> thought it worked on UTF8 strings. Based on this I also tried=20
> out an implementation for the win32-unicode problem that does=20
> *not* require ICU. It uses the win32 native functions to map=20
> to utf16 and back, and then to process the text there. And I=20
> got through with much less code than the ICU version, while=20
> doing the same thing.
>=20
> I am unsure of how to proceed. As I see it there are three paths:
> 1) Use native win32 functionality only on win32
> 2) Use ICU functionality only on win32
> 3) Allow both ICU and native functionality, compile time=20
> switch --with-icu (same as unix with the ICU patch)
>=20
>=20
> The main downsides of ICU vs the native ones are:
> * ICU does not accept win32 locale names. When doing=20
> setlocale("sv_se"), for example, win32 will return this in=20
> later calls as "Swedish_Sweden.1252". To get around this in=20
> the ICU patch, I had to implement a lookup map that converts=20
> it back to sv_se for ICU.
>=20
> * ICU is yet another build and runtime dependency, and a=20
> large one (comes in at 11Mb for the DLL files alone in the=20
> win32 download)
>=20
>=20
> I guess that the main upside of it is that we'd get=20
> constistent behaviour - in case there are issues with either=20
> ICU or win32 native they'd otherwise differ. And only one new=20
> codepath. But we already live with the platform-inconsistency today...
>=20
> Another upside is that it handles more encodings in ICU - my=20
> native implementation does *only* UTF8 and relies on existing=20
> functionality to deal with other encodings. It could of=20
> course be extended if necessary, but from what I can tell=20
> UTF8 is the big one.
>=20
>=20
>=20
> I have attached both patches. For the native version, only=20
> win32_utf8.patch is required.  For the ICU version,=20
> icu_win32.patch is needed and also the files=20
> localemap.c,localemap.pl, iso639 and iso3166 needs to go in=20
> src/backend/port/win32. (the localemap needs to be updated to=20
> do a better-than-linear search, but I wanted to include an example)
>=20
>=20
> Thoughts on the options?
>=20
>=20
> And anohter question - my native patch touches the same=20
> functions as the ICU patch. Can somebody who knows the=20
> internals confirm or deny that these are all the required=20
> locations, or do we need to modify more?
>=20
> (I have run simple tests in swedish locale and both behave=20
> the same and correct, but I'm unsure of exactly how much=20
> would be affected)
>=20
> Finally, the win32 patch also changes the normal path to use=20
> strncoll(). The comment above the function states that we'd=20
> like to use strncoll but it's not available. Well, on win32=20
> it is, so it should provide a speedup on win32. It is=20
> currently not included in the ICU patch, but should probably=20
> be included whichever path we'd chose.
>=20
>=20
> //Magnus
>=20

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="win32_utf8.patch"
Content-Transfer-Encoding: base64
Content-Description: win32_utf8.patch
Content-Disposition: attachment;
	filename="win32_utf8.patch"

SW5kZXg6IHNyYy9iYWNrZW5kL3V0aWxzL2FkdC9vcmFjbGVfY29tcGF0LmMN
Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiAvcHJvamVjdHMv
Y3Zzcm9vdC9wZ3NxbC9zcmMvYmFja2VuZC91dGlscy9hZHQvb3JhY2xlX2Nv
bXBhdC5jLHYNCnJldHJpZXZpbmcgcmV2aXNpb24gMS42MA0KZGlmZiAtYyAt
cjEuNjAgb3JhY2xlX2NvbXBhdC5jDQoqKiogc3JjL2JhY2tlbmQvdXRpbHMv
YWR0L29yYWNsZV9jb21wYXQuYwk3IE1heSAyMDA1IDE1OjE4OjE3IC0wMDAw
CTEuNjANCi0tLSBzcmMvYmFja2VuZC91dGlscy9hZHQvb3JhY2xlX2NvbXBh
dC5jCTMxIEp1bCAyMDA1IDExOjExOjI4IC0wMDAwDQoqKioqKioqKioqKioq
KioNCioqKiAxNDksMTU0ICoqKioNCi0tLSAxNDksMjA5IC0tLS0NCiAgI2Vu
ZGlmICAgLyogVVNFX1dJREVfVVBQRVJfTE9XRVIgKi8NCiAgDQogIA0KKyAj
aWZkZWYgV0lOMzINCisgLyogV2luMzIgVVRGOC9VVEYxNiBjb252ZXJzaW9u
IGhlbHBlcnMgKi8NCisgc3RhdGljIHdjaGFyX3QgKndpbjMyX3V0Zjh0ZXh0
dG91dGYxNihjb25zdCB0ZXh0ICp0eHQpDQorIHsNCisgCWludAkJCW5ieXRl
cyA9IFZBUlNJWkUodHh0KSAtIFZBUkhEUlNaOwkNCisgCXdjaGFyX3QgICAg
KnJlc3VsdDsNCisgCWludCAgICAgICAgIHI7DQorIA0KKyAJaWYgKG5ieXRl
cyA8IDAgfHwNCisgCQkJbmJ5dGVzID4gKGludCkgKElOVF9NQVggLyBzaXpl
b2Yod2NoYXJfdCkpIC0xKQ0KKyAJCWVyZXBvcnQoRVJST1IsDQorIAkJCQko
ZXJyY29kZShFUlJDT0RFX09VVF9PRl9NRU1PUlkpLA0KKyAJCQkJIGVycm1z
Zygib3V0IG9mIG1lbW9yeSIpKSk7DQorIA0KKyAJLyogT3V0cHV0IHdvcmtz
cGFjZSBjYW5ub3QgaGF2ZSBtb3JlIGNvZGVzIHRoYW4gaW5wdXQgYnl0ZXMg
Ki8NCisgCXJlc3VsdCA9ICh3Y2hhcl90ICopIHBhbGxvYygobmJ5dGVzICsg
MSkgKiBzaXplb2Yod2NoYXJfdCkpOw0KKyANCisgCS8qIERvIHRoZSBjb252
ZXJzaW9uICovDQorIAlyID0gTXVsdGlCeXRlVG9XaWRlQ2hhcihDUF9VVEY4
LCAwLCBWQVJEQVRBKHR4dCksICBuYnl0ZXMsDQorIAkJCQlyZXN1bHQsIG5i
eXRlcyAqIHNpemVvZih3Y2hhcl90KSk7DQorIAlpZiAoIXIpDQorIAkJZXJl
cG9ydChFUlJPUiwNCisgCQkJCShlcnJjb2RlKEVSUkNPREVfQ0hBUkFDVEVS
X05PVF9JTl9SRVBFUlRPSVJFKSwNCisgCQkJCSBlcnJtc2coImludmFsaWQg
bXVsdGlieXRlIGNoYXJhY3RlciBmb3IgbG9jYWxlIiksDQorIAkJCQkgZXJy
aGludCgiVGhlIHNlcnZlcidzIExDX0NUWVBFIGxvY2FsZSBpcyBwcm9iYWJs
eSBpbmNvbXBhdGlibGUgd2l0aCB0aGUgZGF0YWJhc2UgZW5jb2RpbmcuIikp
KTsNCisgDQorIAlyZXN1bHRbcl0gPSAwOwkNCisgCXJldHVybiByZXN1bHQ7
DQorIH0NCisgDQorIHN0YXRpYyB0ZXh0ICp3aW4zMl91dGYxNnRvdXRmOHRl
eHQoY29uc3Qgd2NoYXJfdCAqdHh0KQ0KKyB7DQorIAl0ZXh0CQkqcmVzdWx0
Ow0KKyAJaW50CQkJIG5ieXRlczsNCisgCWludAkJCSByOw0KKyAJDQorIAlu
Ynl0ZXMgPSBXaWRlQ2hhclRvTXVsdGlCeXRlKENQX1VURjgsIDAsIHR4dCwg
LTEsIE5VTEwsIDAsIE5VTEwsIE5VTEwpOw0KKyAJaWYgKG5ieXRlcyA9PSAw
KQ0KKyAJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkoZXJyY29kZShFUlJDT0RF
X0NIQVJBQ1RFUl9OT1RfSU5fUkVQRVJUT0lSRSksDQorIAkJCQkgZXJybXNn
KCJpbnZhbGlkIHV0ZjE2IGNoYXJhY3RlciBmb3IgbG9jYWxlOiAlbHUiLEdl
dExhc3RFcnJvcigpKSkpOw0KKyANCisgCXJlc3VsdCA9IHBhbGxvYyhuYnl0
ZXMrVkFSSERSU1opOw0KKyANCisgCXIgPSBXaWRlQ2hhclRvTXVsdGlCeXRl
KENQX1VURjgsIDAsIHR4dCwgLTEsIFZBUkRBVEEocmVzdWx0KSwgbmJ5dGVz
LCBOVUxMLCBOVUxMKTsNCisgCWlmIChyID09IDApDQorIAkJZXJlcG9ydChF
UlJPUiwNCisgCQkJCShlcnJjb2RlKEVSUkNPREVfQ0hBUkFDVEVSX05PVF9J
Tl9SRVBFUlRPSVJFKSwNCisgCQkJCSBlcnJtc2coImludmFsaWQgdXRmMTYg
Y2hhcmFjdGVyIGZvciBsb2NhbGU6ICVsdSIsR2V0TGFzdEVycm9yKCkpKSk7
DQorIA0KKyAJVkFSQVRUX1NJWkVQKHJlc3VsdCkgPSBuYnl0ZXMgKyBWQVJI
RFJTWiAtIDE7IC8qIC0xID0gaWdub3JlIG51bGwgKi8NCisgCXJldHVybiBy
ZXN1bHQ7DQorIH0NCisgI2VuZGlmIC8qIFdJTjMyICovDQorIA0KICAvKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioqKioqKioqKioqKioNCiAgICoNCiAgICogbG93ZXINCioqKioq
KioqKioqKioqKg0KKioqIDE3OSwxODQgKioqKg0KLS0tIDIzNCwyNTEgLS0t
LQ0KICAJCXdjaGFyX3QgICAgKndvcmtzcGFjZTsNCiAgCQlpbnQJCQlpOw0K
ICANCisgI2lmZGVmIFdJTjMyDQorIAkJLyogV2luMzIgZG9lcyBub3QgaGF2
ZSBVVEYtOCwgc28gd2UgbmVlZCB0byBtYXAgdG8gVVRGLTE2ICovDQorIAkJ
aWYgKEdldERhdGFiYXNlRW5jb2RpbmcoKSA9PSBQR19VVEY4KQ0KKyAJCXsN
CisgCQkJd29ya3NwYWNlID0gd2luMzJfdXRmOHRleHR0b3V0ZjE2KHN0cmlu
Zyk7DQorIAkJCV93Y3Nsd3Iod29ya3NwYWNlKTsNCisgCQkJcmVzdWx0ID0g
d2luMzJfdXRmMTZ0b3V0Zjh0ZXh0KHdvcmtzcGFjZSk7DQorIAkJCXBmcmVl
KHdvcmtzcGFjZSk7DQorIAkJCVBHX1JFVFVSTl9URVhUX1AocmVzdWx0KTsN
CisgCQl9DQorICNlbmRpZg0KKyAJCQ0KICAJCXdvcmtzcGFjZSA9IHRleHR0
b3djcyhzdHJpbmcpOw0KICANCiAgCQlmb3IgKGkgPSAwOyB3b3Jrc3BhY2Vb
aV0gIT0gMDsgaSsrKQ0KKioqKioqKioqKioqKioqDQoqKiogMjQ1LDI1MCAq
KioqDQotLS0gMzEyLDMyOSAtLS0tDQogIAkJd2NoYXJfdCAgICAqd29ya3Nw
YWNlOw0KICAJCWludAkJCWk7DQogIA0KKyAjaWZkZWYgV0lOMzINCisgCQkv
KiBXaW4zMiBkb2VzIG5vdCBoYXZlIFVURi04LCBzbyB3ZSBuZWVkIHRvIG1h
cCB0byBVVEYtMTYgKi8NCisgCQlpZiAoR2V0RGF0YWJhc2VFbmNvZGluZygp
ID09IFBHX1VURjgpDQorIAkJew0KKyAJCQl3b3Jrc3BhY2UgPSB3aW4zMl91
dGY4dGV4dHRvdXRmMTYoc3RyaW5nKTsNCisgCQkJX3djc3Vwcih3b3Jrc3Bh
Y2UpOw0KKyAJCQlyZXN1bHQgPSB3aW4zMl91dGYxNnRvdXRmOHRleHQod29y
a3NwYWNlKTsNCisgCQkJcGZyZWUod29ya3NwYWNlKTsNCisgCQkJUEdfUkVU
VVJOX1RFWFRfUChyZXN1bHQpOw0KKyAJCX0NCisgI2VuZGlmDQorIA0KICAJ
CXdvcmtzcGFjZSA9IHRleHR0b3djcyhzdHJpbmcpOw0KICANCiAgCQlmb3Ig
KGkgPSAwOyB3b3Jrc3BhY2VbaV0gIT0gMDsgaSsrKQ0KKioqKioqKioqKioq
KioqDQoqKiogMzE1LDMyMCAqKioqDQotLS0gMzk0LDQwNSAtLS0tDQogIAkJ
aW50CQkJd2FzYWxudW0gPSAwOw0KICAJCWludAkJCWk7DQogIA0KKyAjaWZk
ZWYgV0lOMzINCisgCQkvKiBXaW4zMiBkb2VzIG5vdCBoYXZlIFVURi04LCBz
byB3ZSBuZWVkIHRvIG1hcCB0byBVVEYtMTYgKi8NCisgCQlpZiAoR2V0RGF0
YWJhc2VFbmNvZGluZygpID09IFBHX1VURjgpDQorIAkJCXdvcmtzcGFjZSA9
IHdpbjMyX3V0Zjh0ZXh0dG91dGYxNihzdHJpbmcpOw0KKyAJCWVsc2UNCisg
I2VuZGlmDQogIAkJd29ya3NwYWNlID0gdGV4dHRvd2NzKHN0cmluZyk7DQog
IA0KICAJCWZvciAoaSA9IDA7IHdvcmtzcGFjZVtpXSAhPSAwOyBpKyspDQoq
KioqKioqKioqKioqKioNCioqKiAzMjYsMzMxICoqKioNCi0tLSA0MTEsNDIy
IC0tLS0NCiAgCQkJd2FzYWxudW0gPSBpc3dhbG51bSh3b3Jrc3BhY2VbaV0p
Ow0KICAJCX0NCiAgDQorICNpZmRlZiBXSU4zMg0KKyAJCS8qIE1hcCBiYWNr
IHRvIFVURi04ICovDQorIAkJaWYgKEdldERhdGFiYXNlRW5jb2RpbmcoKSA9
PSBQR19VVEY4KQ0KKyAJCQlyZXN1bHQgPSB3aW4zMl91dGYxNnRvdXRmOHRl
eHQod29ya3NwYWNlKTsNCisgCQllbHNlDQorICNlbmRpZg0KICAJCXJlc3Vs
dCA9IHdjc3RvdGV4dCh3b3Jrc3BhY2UsIGkpOw0KICANCiAgCQlwZnJlZSh3
b3Jrc3BhY2UpOw0KSW5kZXg6IHNyYy9iYWNrZW5kL3V0aWxzL2FkdC92YXJs
ZW5hLmMNCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiAvcHJv
amVjdHMvY3Zzcm9vdC9wZ3NxbC9zcmMvYmFja2VuZC91dGlscy9hZHQvdmFy
bGVuYS5jLHYNCnJldHJpZXZpbmcgcmV2aXNpb24gMS4xMzANCmRpZmYgLWMg
LXIxLjEzMCB2YXJsZW5hLmMNCioqKiBzcmMvYmFja2VuZC91dGlscy9hZHQv
dmFybGVuYS5jCTI5IEp1bCAyMDA1IDAzOjE3OjU1IC0wMDAwCTEuMTMwDQot
LS0gc3JjL2JhY2tlbmQvdXRpbHMvYWR0L3ZhcmxlbmEuYwkzMSBKdWwgMjAw
NSAxMToxNTo0OSAtMDAwMA0KKioqKioqKioqKioqKioqDQoqKiogODQ4LDg1
MyAqKioqDQotLS0gODQ4LDkwMSAtLS0tDQogIAkJY2hhcgkJYTJidWZbU1RB
Q0tCVUZMRU5dOw0KICAJCWNoYXIJICAgKmExcCwNCiAgCQkJCSAgICphMnA7
DQorICNpZmRlZiBXSU4zMg0KKyAJCS8qIFdpbjMyIGRvZXMgbm90IGhhdmUg
VVRGLTgsIHNvIHdlIG5lZWQgdG8gbWFwIHRvIFVURi0xNiAqLw0KKyAJCWlm
IChHZXREYXRhYmFzZUVuY29kaW5nKCkgPT0gUEdfVVRGOCkNCisgCQl7DQor
IAkJCWludCBhMWxlbiA9IFNUQUNLQlVGTEVOOw0KKyAJCQlpbnQgYTJsZW4g
PSBTVEFDS0JVRkxFTjsNCisgDQorIAkJCWlmIChsZW4xID49IFNUQUNLQlVG
TEVOLzIpDQorIAkJCXsNCisgCQkJCWExbGVuID0gbGVuMSAqIDIgKyAyOw0K
KyAJCQkJYTFwID0gcGFsbG9jKGExbGVuKTsNCisgCQkJfQ0KKyAJCQllbHNl
DQorIAkJCQlhMXAgPSBhMWJ1ZjsNCisgCQkJDQorIAkJCWlmIChsZW4yID49
IFNUQUNLQlVGTEVOLzIpDQorIAkJCXsNCisgCQkJCWEybGVuID0gbGVuMiAq
IDIgKyAyOw0KKyAJCQkJYTJwID0gcGFsbG9jKGEybGVuKTsNCisgCQkJfQ0K
KyAJCQllbHNlDQorIAkJCQlhMnAgPSBhMmJ1ZjsNCisgDQorIAkJCWlmICgh
TXVsdGlCeXRlVG9XaWRlQ2hhcihDUF9VVEY4LCAwLCBhcmcxLCAtMSwgKExQ
V1NUUilhMXAsIGExbGVuLzIpKQ0KKyAJCQkJZXJlcG9ydChFUlJPUiwNCisg
CQkJCQkJKGVycm1zZygiZmFpbGVkIHRvIGNvbnZlcnQgc3RyaW5nIHRvIFVU
RjE2OiAlbHUiLA0KKyAJCQkJCQkJR2V0TGFzdEVycm9yKCkpKSk7DQorIAkJ
CWlmICghTXVsdGlCeXRlVG9XaWRlQ2hhcihDUF9VVEY4LCAwLCBhcmcyLCAt
MSwgKExQV1NUUilhMnAsIGEybGVuLzIpKQ0KKyAJCQkJZXJlcG9ydChFUlJP
UiwNCisgCQkJCQkJKGVycm1zZygiZmFpbGVkIHRvIGNvbnZlcnQgc3RyaW5n
IHRvIFVURjE2OiAlbHUiLA0KKyAJCQkJCQkJR2V0TGFzdEVycm9yKCkpKSk7
DQorIAkJCQ0KKyAJCQllcnJubz0wOw0KKyAJCQlyZXN1bHQgPSB3Y3Njb2xs
KChMUFdTVFIpYTFwLCAoTFBXU1RSKWEycCk7DQorIAkJCWlmIChyZXN1bHQg
PT0gMjE0NzQ4MzY0NykgLyogX05MU0NNUEVSUk9SOyBtaXNzaW5nIGZyb20g
bWluZ3cgaGVhZGVycyAqLw0KKyAJCQkJZXJlcG9ydChFUlJPUiwNCisgCQkJ
CQkJKGVycm1zZygiZmFpbGVkIHRvIGNvbXBhcmUgdW5pY29kZSBzdHJpbmdz
OiAlaSIsIGVycm5vKSkpOw0KKyAJCQlpZiAoYTFsZW4gIT0gU1RBQ0tCVUZM
RU4pDQorIAkJCQlwZnJlZShhMXApOw0KKyAJCQlpZiAoYTJsZW4gIT0gU1RB
Q0tCVUZMRU4pDQorIAkJCQlwZnJlZShhMnApOw0KKyAJCQkNCisgCQkJcmV0
dXJuIHJlc3VsdDsNCisgCQl9DQorIA0KKyAJCS8qIFdpbjMyIGhhcyBzdHJu
Y29sbCgpLCBzbyB1c2UgaXQgdGhlcmUgdG8gYXZvaWQgY29weWluZyAqLw0K
KyAJCXJldHVybiBfc3RybmNvbGwoYXJnMSwgYXJnMiwgKGxlbjE+bGVuMik/
bGVuMjpsZW4xICk7DQorICNlbmRpZg0KICANCiAgCQlpZiAobGVuMSA+PSBT
VEFDS0JVRkxFTikNCiAgCQkJYTFwID0gKGNoYXIgKikgcGFsbG9jKGxlbjEg
KyAxKTsNCg==

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="icu_win32.patch"
Content-Transfer-Encoding: base64
Content-Description: icu_win32.patch
Content-Disposition: attachment;
	filename="icu_win32.patch"

SW5kZXg6IGNvbmZpZ3VyZS5pbg0KPT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0K
UkNTIGZpbGU6IC9wcm9qZWN0cy9jdnNyb290L3Bnc3FsL2NvbmZpZ3VyZS5p
bix2DQpyZXRyaWV2aW5nIHJldmlzaW9uIDEuNDE3DQpkaWZmIC1jIC1yMS40
MTcgY29uZmlndXJlLmluDQoqKiogY29uZmlndXJlLmluCTYgSnVsIDIwMDUg
MjE6MDQ6MTMgLTAwMDAJMS40MTcNCi0tLSBjb25maWd1cmUuaW4JMjggSnVs
IDIwMDUgMTc6MDE6NDcgLTAwMDANCioqKioqKioqKioqKioqKg0KKioqIDQ2
Nyw0NzIgKioqKg0KLS0tIDQ2Nyw0ODAgLS0tLQ0KICBBQ19NU0dfUkVTVUxU
KFskd2l0aF9vcGVuc3NsXSkNCiAgQUNfU1VCU1Qod2l0aF9vcGVuc3NsKQ0K
ICANCisgIw0KKyAjIElDVQ0KKyAjDQorIEFDX01TR19DSEVDS0lORyhbd2hl
dGhlciB0byBidWlsZCB3aXRoIElDVSBzdXBwb3J0XSkNCisgUEdBQ19BUkdf
Qk9PTCh3aXRoLCBpY3UsIG5vLCBbICAtLXdpdGgtaWN1ICAgICAgICAgICAg
ICBidWlsZCB3aXRoIElDVSBzdXBwb3J0XSwNCisgICAgICAgICAgICAgICBb
QUNfREVGSU5FKFtVU0VfSUNVXSwgMSwgW0RlZmluZSB0byBidWlsZCB3aXRo
IElDVSBzdXBwb3J0LiAoLS13aXRoLWljdSldKV0pDQorIEFDX01TR19SRVNV
TFQoWyR3aXRoX2ljdV0pDQorIEFDX1NVQlNUKHdpdGhfaWN1KQ0KICANCiAg
Iw0KICAjIFJlYWRsaW5lDQoqKioqKioqKioqKioqKioNCioqKiA2NzQsNjc5
ICoqKioNCi0tLSA2ODIsNjk4IC0tLS0NCiAgICBmaQ0KICBmaQ0KICANCisg
aWYgdGVzdCAiJHdpdGhfaWN1IiA9IHllcyA7IHRoZW4NCisgICBpZiB0ZXN0
ICIkUE9SVE5BTUUiICE9ICJ3aW4zMiI7IHRoZW4NCisgICAgIEFDX0NIRUNL
X0xJQihpY3VpMThuLCB1Y29sX29wZW5fM18yLCBbXSwgW0FDX01TR19FUlJP
UihbbGlicmFyeSAnaWN1aTE4bicgaXMgcmVxdWlyZWQgZm9yIElDVV0pXSkN
CisgICAgIEFDX0NIRUNLX0xJQihpY3V1YywgIHVfdG9sb3dlcl8zXzIsIFtd
LCBbQUNfTVNHX0VSUk9SKFtsaWJyYXJ5ICdpY3V1YycgaXMgcmVxdWlyZWQg
Zm9yIElDVV0pXSkNCisgICAgIEFDX0NIRUNLX0xJQihpY3VkYXRhLCBpY3Vk
dDMyX2RhdCwgW10sIFtBQ19NU0dfRVJST1IoW2xpYnJhcnkgJ2ljdWRhdGEn
IGlzIHJlcXVpcmVkIGZvciBJQ1VdKV0pDQorICAgZWxzZQ0KKyAgICAgQUNf
Q0hFQ0tfTElCKGljdWluLCB1Y29sX29wZW5fM18yLCBbXSwgW0FDX01TR19F
UlJPUihbbGlicmFyeSAnaWN1aW4nIGlzIHJlcXVpcmVkIGZvciBJQ1VdKV0p
DQorICAgICBBQ19DSEVDS19MSUIoaWN1dWMsICB1X3RvbG93ZXJfM18yLCBb
XSwgW0FDX01TR19FUlJPUihbbGlicmFyeSAnaWN1dWMnIGlzIHJlcXVpcmVk
IGZvciBJQ1VdKV0pDQorICAgZmkNCisgZmkNCisgICANCiAgaWYgdGVzdCAi
JHdpdGhfcGFtIiA9IHllcyA7IHRoZW4NCiAgICBBQ19DSEVDS19MSUIocGFt
LCAgICBwYW1fc3RhcnQsIFtdLCBbQUNfTVNHX0VSUk9SKFtsaWJyYXJ5ICdw
YW0nIGlzIHJlcXVpcmVkIGZvciBQQU1dKV0pDQogIGZpDQoqKioqKioqKioq
KioqKioNCioqKiA3NDgsNzUzICoqKioNCi0tLSA3NjcsNzc2IC0tLS0NCiAg
ICBBQ19DSEVDS19IRUFERVIob3BlbnNzbC9lcnIuaCwgW10sIFtBQ19NU0df
RVJST1IoW2hlYWRlciBmaWxlIDxvcGVuc3NsL2Vyci5oPiBpcyByZXF1aXJl
ZCBmb3IgT3BlblNTTF0pXSkNCiAgZmkNCiAgDQorIGlmIHRlc3QgIiR3aXRo
X2ljdSIgPSB5ZXMgOyB0aGVuDQorICAgQUNfQ0hFQ0tfSEVBREVSKHVuaWNv
ZGUvdXR5cGVzLmgsIFtdLCBbQUNfTVNHX0VSUk9SKFtoZWFkZXIgZmlsZSA8
dW5pY29kZS91dHlwZXMuaD4gaXMgcmVxdWlyZWQgZm9yIElDVV0pXSkNCisg
ZmkNCisgIA0KICBpZiB0ZXN0ICIkd2l0aF9wYW0iID0geWVzIDsgdGhlbg0K
ICAgIEFDX0NIRUNLX0hFQURFUlMoc2VjdXJpdHkvcGFtX2FwcGwuaCwgW10s
DQogICAgICAgICAgICAgICAgICAgICBbQUNfQ0hFQ0tfSEVBREVSUyhwYW0v
cGFtX2FwcGwuaCwgW10sDQpJbmRleDogc3JjL2JhY2tlbmQvcG9ydC93aW4z
Mi9NYWtlZmlsZQ0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KUkNTIGZpbGU6
IC9wcm9qZWN0cy9jdnNyb290L3Bnc3FsL3NyYy9iYWNrZW5kL3BvcnQvd2lu
MzIvTWFrZWZpbGUsdg0KcmV0cmlldmluZyByZXZpc2lvbiAxLjYNCmRpZmYg
LWMgLXIxLjYgTWFrZWZpbGUNCioqKiBzcmMvYmFja2VuZC9wb3J0L3dpbjMy
L01ha2VmaWxlCTI5IEF1ZyAyMDA0IDAwOjM4OjAzIC0wMDAwCTEuNg0KLS0t
IHNyYy9iYWNrZW5kL3BvcnQvd2luMzIvTWFrZWZpbGUJMzAgSnVsIDIwMDUg
MTc6NTI6NDcgLTAwMDANCioqKioqKioqKioqKioqKg0KKioqIDEyLDI5ICoq
KioNCiAgdG9wX2J1aWxkZGlyID0gLi4vLi4vLi4vLi4NCiAgaW5jbHVkZSAk
KHRvcF9idWlsZGRpcikvc3JjL01ha2VmaWxlLmdsb2JhbA0KICANCiEgT0JK
UyA9IHNlbWEubyBzaG1lbS5vIHRpbWVyLm8gc29ja2V0Lm8gc2lnbmFsLm8g
c2VjdXJpdHkubyBlcnJvci5vDQogIA0KICBhbGw6IFNVQlNZUy5vDQogIA0K
ICBTVUJTWVMubzogJChPQkpTKQ0KICAJJChMRCkgJChMRFJFTCkgJChMRE9V
VCkgU1VCU1lTLm8gJChPQkpTKQ0KICANCiAgZGVwZW5kIGRlcDoNCiAgCSQo
Q0MpIC1NTSAkKENGTEFHUykgKi5jID5kZXBlbmQNCiAgDQogIGNsZWFuOiAN
CiEgCXJtIC1mIFNVQlNZUy5vICQoT0JKUykNCiAgDQogIGlmZXEgKGRlcGVu
ZCwkKHdpbGRjYXJkIGRlcGVuZCkpDQogIGluY2x1ZGUgZGVwZW5kDQotLS0g
MTIsMzQgLS0tLQ0KICB0b3BfYnVpbGRkaXIgPSAuLi8uLi8uLi8uLg0KICBp
bmNsdWRlICQodG9wX2J1aWxkZGlyKS9zcmMvTWFrZWZpbGUuZ2xvYmFsDQog
IA0KISBPQkpTID0gc2VtYS5vIHNobWVtLm8gdGltZXIubyBzb2NrZXQubyBz
aWduYWwubyBzZWN1cml0eS5vIGVycm9yLm8gbG9jYWxlbWFwLm8NCiAgDQog
IGFsbDogU1VCU1lTLm8NCiAgDQogIFNVQlNZUy5vOiAkKE9CSlMpDQogIAkk
KExEKSAkKExEUkVMKSAkKExET1VUKSBTVUJTWVMubyAkKE9CSlMpDQogIA0K
KyBsb2NhbGVtYXAubzogbG9jYWxlbWFwLmMgbG9jYWxlbWFwLmgNCisgDQor
IGxvY2FsZW1hcC5oOiBpc282MzkgaXNvMzE2NiBsb2NhbGVtYXAucGwNCisg
CSQoUEVSTCkgbG9jYWxlbWFwLnBsID4gbG9jYWxlbWFwLmgNCisgDQogIGRl
cGVuZCBkZXA6DQogIAkkKENDKSAtTU0gJChDRkxBR1MpICouYyA+ZGVwZW5k
DQogIA0KICBjbGVhbjogDQohIAlybSAtZiBTVUJTWVMubyAkKE9CSlMpIGxv
Y2FsZW1hcC5oDQogIA0KICBpZmVxIChkZXBlbmQsJCh3aWxkY2FyZCBkZXBl
bmQpKQ0KICBpbmNsdWRlIGRlcGVuZA0KSW5kZXg6IHNyYy9iYWNrZW5kL3V0
aWxzL2FkdC9vcmFjbGVfY29tcGF0LmMNCj09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT0NClJDUyBmaWxlOiAvcHJvamVjdHMvY3Zzcm9vdC9wZ3NxbC9zcmMvYmFj
a2VuZC91dGlscy9hZHQvb3JhY2xlX2NvbXBhdC5jLHYNCnJldHJpZXZpbmcg
cmV2aXNpb24gMS42MA0KZGlmZiAtYyAtcjEuNjAgb3JhY2xlX2NvbXBhdC5j
DQoqKiogc3JjL2JhY2tlbmQvdXRpbHMvYWR0L29yYWNsZV9jb21wYXQuYwk3
IE1heSAyMDA1IDE1OjE4OjE3IC0wMDAwCTEuNjANCi0tLSBzcmMvYmFja2Vu
ZC91dGlscy9hZHQvb3JhY2xlX2NvbXBhdC5jCTMwIEp1bCAyMDA1IDE0OjU5
OjI5IC0wMDAwDQoqKioqKioqKioqKioqKioNCioqKiAzMiwzNyAqKioqDQot
LS0gMzIsNDMgLS0tLQ0KICAjaW5jbHVkZSAidXRpbHMvcGdfbG9jYWxlLmgi
DQogICNpbmNsdWRlICJtYi9wZ193Y2hhci5oIg0KICANCisgI2lmZGVmIFVT
RV9JQ1UNCisgI2luY2x1ZGUgPHVuaWNvZGUvdXR5cGVzLmg+ICAgLyogQmFz
aWMgSUNVIGRhdGEgdHlwZXMgKi8NCisgI2luY2x1ZGUgPHVuaWNvZGUvdWNu
di5oPiAgICAgLyogQyAgIENvbnZlcnRlciBBUEkgICAgKi8NCisgI2luY2x1
ZGUgPHVuaWNvZGUvdXN0cmluZy5oPg0KKyAjZW5kaWYgLyogVVNFX0lDVSAq
Lw0KKyANCiAgDQogIC8qDQogICAqIElmIHRoZSBzeXN0ZW0gcHJvdmlkZXMg
dGhlIG5lZWRlZCBmdW5jdGlvbnMgZm9yIHdpZGUtY2hhcmFjdGVyIG1hbmlw
dWxhdGlvbg0KKioqKioqKioqKioqKioqDQoqKiogNTMsNTggKioqKg0KLS0t
IDU5LDEwNiAtLS0tDQogIAkgICBib29sIGRvbHRyaW0sIGJvb2wgZG9ydHJp
bSk7DQogIA0KICANCisgI2lmZGVmIFVTRV9JQ1UNCisgc3RhdGljIFVDb252
ZXJ0ZXIgKmNvbnYgPSBOVUxMOw0KKyANCisgc3RhdGljIHRleHQgKg0KKyBV
Q2hhcnRvdGV4dChjb25zdCBVQ2hhciAqc3RyLCBpbnQgbmNvZGVzKQ0KKyB7
DQorIAl0ZXh0CSAgICpyZXN1bHQ7DQorIAlzaXplX3QJCW5ieXRlcywgcmVz
dWx0c2l6ZTsNCisgDQorIAlVRXJyb3JDb2RlIHN0YXR1cyA9IFVfWkVST19F
UlJPUjsNCisgDQorIAkvKiBPdmVyZmxvdyBwYXJhbm9pYSAqLw0KKyAJaWYg
KG5jb2RlcyA8IDAgfHwNCisgCQluY29kZXMgPiAoaW50KSAoKElOVF9NQVgg
LSBWQVJIRFJTWikgLyBzaXplb2YoVUNoYXIpKSAtIDEpDQorIAkJZXJlcG9y
dChFUlJPUiwNCisgCQkJCShlcnJjb2RlKEVSUkNPREVfT1VUX09GX01FTU9S
WSksDQorIAkJCQkgZXJybXNnKCJvdXQgb2YgbWVtb3J5IikpKTsNCisgDQor
IAkvKiBNYWtlIHdvcmtzcGFjZSBjZXJ0YWlubHkgbGFyZ2UgZW5vdWdoIGZv
ciByZXN1bHQgKi8NCisgCXJlc3VsdHNpemUgPSBVQ05WX0dFVF9NQVhfQllU
RVNfRk9SX1NUUklORyhuY29kZXMsIHVjbnZfZ2V0TWF4Q2hhclNpemUoY29u
dikpOw0KKyAJcmVzdWx0ID0gKHRleHQgKikgcGFsbG9jKHJlc3VsdHNpemUg
KyBWQVJIRFJTWik7DQorIA0KKyAJbmJ5dGVzID0gdWNudl9mcm9tVUNoYXJz
KGNvbnYsIFZBUkRBVEEocmVzdWx0KSwgcmVzdWx0c2l6ZSwNCisgCQkJCQkJ
CSBzdHIsIG5jb2RlcywgJnN0YXR1cyk7DQorIA0KKyAJaWYgKFVfRkFJTFVS
RShzdGF0dXMpKQ0KKyAJew0KKyAJCS8qIEludmFsaWQgbXVsdGlieXRlIGNo
YXJhY3RlciBlbmNvdW50ZXJlZCAuLi4gc2hvdWxkbid0IGhhcHBlbiAqLw0K
KyAJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkoZXJyY29kZShFUlJDT0RFX0NI
QVJBQ1RFUl9OT1RfSU5fUkVQRVJUT0lSRSksDQorIAkJCQkgZXJybXNnKCJp
bnZhbGlkIG11bHRpYnl0ZSBjaGFyYWN0ZXIgZm9yIGxvY2FsZSIpKSk7DQor
IAl9DQorIA0KKyAJQXNzZXJ0KG5ieXRlcyA8PSAoc2l6ZV90KSAobmNvZGVz
ICogc2l6ZW9mKFVDaGFyKSkpOw0KKyANCisgCVZBUkFUVF9TSVpFUChyZXN1
bHQpID0gbmJ5dGVzICsgVkFSSERSU1o7DQorIA0KKyAJcmV0dXJuIHJlc3Vs
dDsNCisgfQ0KKyANCisgDQorICNlbHNlDQogICNpZmRlZiBVU0VfV0lERV9V
UFBFUl9MT1dFUg0KICANCiAgLyoNCioqKioqKioqKioqKioqKg0KKioqIDE0
NywxNTIgKioqKg0KLS0tIDE5NSwyMDEgLS0tLQ0KICAJcmV0dXJuIHJlc3Vs
dDsNCiAgfQ0KICAjZW5kaWYgICAvKiBVU0VfV0lERV9VUFBFUl9MT1dFUiAq
Lw0KKyAjZW5kaWYgICAvKiBVU0VfSUNVICovDQogIA0KICANCiAgLyoqKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioqKioqKioqKioqDQoqKioqKioqKioqKioqKioNCioqKiAxNjYs
MTcxICoqKioNCi0tLSAyMTUsMjg2IC0tLS0NCiAgRGF0dW0NCiAgbG93ZXIo
UEdfRlVOQ1RJT05fQVJHUykNCiAgew0KKyAjaWZkZWYgVVNFX0lDVQ0KKyAj
ZGVmaW5lIFNUQUNLQlVGTEVOCQkxMDI0IC8gc2l6ZW9mKFVDaGFyKQ0KKyAJ
LyogdXNlIElDVSBvbmx5IHdoZW4gbWF4IGVuY29kaW5nIGxlbmd0aCA+IG9u
ZSAqLw0KKyAJaWYgKHBnX2RhdGFiYXNlX2VuY29kaW5nX21heF9sZW5ndGgo
KSA+IDEpDQorIAl7DQorIAkJdGV4dAkgICAqc3RyaW5nID0gUEdfR0VUQVJH
X1RFWFRfUCgwKTsNCisgCQl0ZXh0CSAgICpyZXN1bHQ7DQorIAkJVUNoYXIg
ICAgICAgc291cmNlYnVmW1NUQUNLQlVGTEVOXSwgZGVzdGJ1ZltTVEFDS0JV
RkxFTl07DQorIAkJVUNoYXIgICAgICAqc291cmNlLCAqZGVzdDsNCisgCQlp
bnQJCQlidWZsZW4sDQorIAkJCQkJYXJnbGVuID0gVkFSU0laRShzdHJpbmcp
IC0gVkFSSERSU1o7DQorIAkJVUVycm9yQ29kZSAgc3RhdHVzID0gVV9aRVJP
X0VSUk9SOw0KKyANCisgCQlpZiAoY29udiA9PSBOVUxMKQ0KKyAJCXsNCisg
CQkJY29udiA9IHVjbnZfb3BlbihOVUxMLCAmc3RhdHVzKTsNCisgCQkJaWYg
KFVfRkFJTFVSRShzdGF0dXMpKQ0KKyAJCQl7DQorIAkJCQllcmVwb3J0KEVS
Uk9SLA0KKyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJy
bXNnKCJJQ1UgZXJyb3I6IG9yYWNsZV9jb21wYXQuYywgY291bGQgbm90IGdl
dCBjb252ZXJ0ZXIgZm9yIFwiJXNcIiIsIHVjbnZfZ2V0RGVmYXVsdE5hbWUo
KSkpKTsNCisgCQkJfQ0KKyAJCX0NCisgDQorIAkJaWYgKGFyZ2xlbiA+PSBT
VEFDS0JVRkxFTiAvIHNpemVvZihVQ2hhcikpDQorIAkJew0KKyAJCQlidWZs
ZW4gPSBhcmdsZW4gKiBzaXplb2YoVUNoYXIpICsgMTsNCisgIAkJCXNvdXJj
ZSA9IHBhbGxvYyhidWZsZW4pOw0KKyAJCQlkZXN0ID0gcGFsbG9jKGJ1Zmxl
bik7DQorIAkJfQ0KKyAJCWVsc2UNCisgCQl7DQorIAkJCWJ1ZmxlbiA9IFNU
QUNLQlVGTEVOOw0KKyAJCQlzb3VyY2UgPSBzb3VyY2VidWY7DQorIAkJCWRl
c3QgPSBkZXN0YnVmOw0KKyAJCX0NCisgCQkvLyBjb252ZXJ0IHRvIFVURi0x
Ng0KKyAJCXVjbnZfdG9VQ2hhcnMoY29udiwgc291cmNlLCBidWZsZW4sIFZB
UkRBVEEoc3RyaW5nKSwgYXJnbGVuLCAmc3RhdHVzKTsNCisgCQlpZiAoVV9G
QUlMVVJFKHN0YXR1cykpDQorIAkJew0KKyAJCQllcmVwb3J0KEVSUk9SLA0K
KyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJ
Q1UgZXJyb3I6IENvdWxkIG5vdCBjb252ZXJ0IHN0cmluZyIpKSk7DQorIAkJ
fQ0KKyAJCQ0KKyAJCS8vIHJ1biBkZXNpcmVkIGZ1bmN0aW9uDQorIAkJYnVm
bGVuID0gdV9zdHJUb0xvd2VyKGRlc3QsIGJ1Zmxlbiwgc291cmNlLCAtMSwg
TlVMTCwgJnN0YXR1cyk7DQorIAkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K
KyAJCXsNCisgCQkJZXJlcG9ydChFUlJPUiwNCisgCQkJCQkJKGVycmNvZGUo
c3RhdHVzKSwNCisgCQkJCQkJIGVycm1zZygiSUNVIGVycm9yOiBDb3VsZCBu
b3QgbW9kaWZ5IGNhc2UiKSkpOw0KKyAJCX0NCisgDQorIAkJLy8gYW5kIGNv
bnZlcnQgbW9kaWZpZWQgdXRmLTE2IHN0cmluZyBiYWNrIHRvIHRleHQNCisg
CQlyZXN1bHQgPSBVQ2hhcnRvdGV4dChkZXN0LCBidWZsZW4pOw0KKyANCisg
CQlpZiAoYXJnbGVuID49IFNUQUNLQlVGTEVOIC8gc2l6ZW9mKFVDaGFyKSkN
CisgCQl7DQorIAkJCXBmcmVlKHNvdXJjZSk7DQorIAkJCXBmcmVlKGRlc3Qp
Ow0KKyAJCX0NCisgCQlQR19SRVRVUk5fVEVYVF9QKHJlc3VsdCk7DQorIAl9
DQorIAllbHNlDQorICNlbHNlDQogICNpZmRlZiBVU0VfV0lERV9VUFBFUl9M
T1dFUg0KICAJLyoNCiAgCSAqCVVzZSB3aWRlIGNoYXIgY29kZSBvbmx5IHdo
ZW4gbWF4IGVuY29kaW5nIGxlbmd0aCA+IDEgYW5kIGN0eXBlICE9IEMuDQoq
KioqKioqKioqKioqKioNCioqKiAxOTIsMTk3ICoqKioNCi0tLSAzMDcsMzEz
IC0tLS0NCiAgCX0NCiAgCWVsc2UNCiAgI2VuZGlmICAgLyogVVNFX1dJREVf
VVBQRVJfTE9XRVIgKi8NCisgI2VuZGlmICAgLyogVVNFX0lDVSAqLw0KICAJ
ew0KICAJCXRleHQJICAgKnN0cmluZyA9IFBHX0dFVEFSR19URVhUX1BfQ09Q
WSgwKTsNCiAgCQljaGFyCSAgICpwdHI7DQoqKioqKioqKioqKioqKioNCioq
KiAyMzIsMjM3ICoqKioNCi0tLSAzNDgsNDE4IC0tLS0NCiAgRGF0dW0NCiAg
dXBwZXIoUEdfRlVOQ1RJT05fQVJHUykNCiAgew0KKyAjaWZkZWYgVVNFX0lD
VQ0KKyAJLyogdXNlIElDVSBvbmx5IHdoZW4gbWF4IGVuY29kaW5nIGxlbmd0
aCA+IG9uZSAqLw0KKyAJaWYgKHBnX2RhdGFiYXNlX2VuY29kaW5nX21heF9s
ZW5ndGgoKSA+IDEpDQorIAl7DQorIAkJdGV4dAkgICAqc3RyaW5nID0gUEdf
R0VUQVJHX1RFWFRfUCgwKTsNCisgCQl0ZXh0CSAgICpyZXN1bHQ7DQorIAkJ
VUNoYXIgICAgICAgc291cmNlYnVmW1NUQUNLQlVGTEVOXSwgZGVzdGJ1ZltT
VEFDS0JVRkxFTl07DQorIAkJVUNoYXIgICAgICAqc291cmNlLCAqZGVzdDsN
CisgCQlpbnQzMl90ICAgICBidWZsZW4sIGFyZ2xlbjsNCisgDQorIAkJVUVy
cm9yQ29kZSAgc3RhdHVzID0gVV9aRVJPX0VSUk9SOw0KKyANCisgCQlpZiAo
Y29udiA9PSBOVUxMKQ0KKyAJCXsNCisgCQkJY29udiA9IHVjbnZfb3BlbihO
VUxMLCAmc3RhdHVzKTsNCisgCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K
KyAJCQl7DQorIAkJCQllcmVwb3J0KEVSUk9SLA0KKyAJCQkJCQkoZXJyY29k
ZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJQ1UgZXJyb3I6IENvdWxk
IG5vdCBnZXQgY29udmVydGVyIGZvciBcIiVzXCIiLCB1Y252X2dldERlZmF1
bHROYW1lKCkpKSk7DQorIAkJCX0NCisgCQl9DQorIAkJYXJnbGVuID0gVkFS
U0laRShzdHJpbmcpIC0gVkFSSERSU1o7DQorIAkJaWYgKGFyZ2xlbiAqIHNp
emVvZihVQ2hhcikgPj0gU1RBQ0tCVUZMRU4pDQorIAkJew0KKyAJCQlidWZs
ZW4gPSBhcmdsZW4gKiBzaXplb2YoVUNoYXIpICsgMTsNCisgIAkJCXNvdXJj
ZSA9IHBhbGxvYyhidWZsZW4pOw0KKyAJCQlkZXN0ID0gcGFsbG9jKGJ1Zmxl
bik7DQorIAkJfQ0KKyAJCWVsc2UNCisgCQl7DQorIAkJCWJ1ZmxlbiA9IFNU
QUNLQlVGTEVOOw0KKyAJCQlzb3VyY2UgPSBzb3VyY2VidWY7DQorIAkJCWRl
c3QgPSBkZXN0YnVmOw0KKyAJCX0NCisgCQkvLyBjb252ZXJ0IHRvIFVURi0x
Ng0KKyAJCXVjbnZfdG9VQ2hhcnMoY29udiwgc291cmNlLCBidWZsZW4sIFZB
UkRBVEEoc3RyaW5nKSwgYXJnbGVuLCAmc3RhdHVzKTsNCisgCQlpZiAoVV9G
QUlMVVJFKHN0YXR1cykpDQorIAkJew0KKyAJCQllcmVwb3J0KEVSUk9SLA0K
KyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJ
Q1UgZXJyb3I6IENvdWxkIG5vdCBjb252ZXJ0IHN0cmluZyIpKSk7DQorIAkJ
fQ0KKyANCisgCQkvLyBydW4gZGVzaXJlZCBmdW5jdGlvbg0KKyAJCWJ1Zmxl
biA9IHVfc3RyVG9VcHBlcihkZXN0LCBidWZsZW4sIHNvdXJjZSwgLTEsIE5V
TEwsICZzdGF0dXMpOw0KKyAJCWlmIChVX0ZBSUxVUkUoc3RhdHVzKSkNCisg
CQl7DQorIAkJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkJCShlcnJjb2RlKHN0
YXR1cyksDQorIAkJCQkJCSBlcnJtc2coIklDVSBlcnJvcjogQ291bGQgbm90
IG1vZGlmeSBjYXNlIikpKTsNCisgCQl9DQorIA0KKyAJCS8vIGFuZCBjb252
ZXJ0IG1vZGlmaWVkIHV0Zi0xNiBzdHJpbmcgYmFjayB0byB0ZXh0DQorIAkJ
cmVzdWx0ID0gVUNoYXJ0b3RleHQoZGVzdCwgYnVmbGVuKTsNCisgDQorIAkJ
aWYgKGFyZ2xlbiAqIHNpemVvZihVQ2hhcikgPj0gU1RBQ0tCVUZMRU4pDQor
IAkJew0KKyAJCQlwZnJlZShzb3VyY2UpOw0KKyAJCQlwZnJlZShkZXN0KTsN
CisgCQl9DQorIAkJUEdfUkVUVVJOX1RFWFRfUChyZXN1bHQpOw0KKyAJfQ0K
KyAJZWxzZQ0KKyAjZWxzZQ0KICAjaWZkZWYgVVNFX1dJREVfVVBQRVJfTE9X
RVINCiAgCS8qDQogIAkgKglVc2Ugd2lkZSBjaGFyIGNvZGUgb25seSB3aGVu
IG1heCBlbmNvZGluZyBsZW5ndGggPiAxIGFuZCBjdHlwZSAhPSBDLg0KKioq
KioqKioqKioqKioqDQoqKiogMjU4LDI2MyAqKioqDQotLS0gNDM5LDQ0NSAt
LS0tDQogIAl9DQogIAllbHNlDQogICNlbmRpZiAgIC8qIFVTRV9XSURFX1VQ
UEVSX0xPV0VSICovDQorICNlbmRpZiAgIC8qIFVTRV9JQ1UgKi8NCiAgCXsN
CiAgCQl0ZXh0CSAgICpzdHJpbmcgPSBQR19HRVRBUkdfVEVYVF9QX0NPUFko
MCk7DQogIAkJY2hhcgkgICAqcHRyOw0KKioqKioqKioqKioqKioqDQoqKiog
MzAxLDMwNiAqKioqDQotLS0gNDgzLDU1MyAtLS0tDQogIERhdHVtDQogIGlu
aXRjYXAoUEdfRlVOQ1RJT05fQVJHUykNCiAgew0KKyAjaWZkZWYgVVNFX0lD
VQ0KKyAJLyogdXNlIElDVSBvbmx5IHdoZW4gbWF4IGVuY29kaW5nIGxlbmd0
aCA+IG9uZSAqLw0KKyAJaWYgKHBnX2RhdGFiYXNlX2VuY29kaW5nX21heF9s
ZW5ndGgoKSA+IDEpDQorIAl7DQorIAkJdGV4dAkgICAqc3RyaW5nID0gUEdf
R0VUQVJHX1RFWFRfUCgwKTsNCisgCQl0ZXh0CSAgICpyZXN1bHQ7DQorIAkJ
VUNoYXIgICAgICAgc291cmNlYnVmW1NUQUNLQlVGTEVOXSwgZGVzdGJ1ZltT
VEFDS0JVRkxFTl07DQorIAkJVUNoYXIgICAgICAqc291cmNlLCAqZGVzdDsN
CisgCQlpbnQzMl90ICAgICBidWZsZW4sIGFyZ2xlbjsNCisgDQorIAkJVUVy
cm9yQ29kZSAgc3RhdHVzID0gVV9aRVJPX0VSUk9SOw0KKyANCisgCQlpZiAo
Y29udiA9PSBOVUxMKQ0KKyAJCXsNCisgCQkJY29udiA9IHVjbnZfb3BlbihO
VUxMLCAmc3RhdHVzKTsNCisgCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K
KyAJCQl7DQorIAkJCQllcmVwb3J0KEVSUk9SLA0KKyAJCQkJCQkoZXJyY29k
ZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJQ1UgZXJyb3I6IENvdWxk
IG5vdCBnZXQgY29udmVydGVyIGZvciBcIiVzXCIiLCB1Y252X2dldERlZmF1
bHROYW1lKCkpKSk7DQorIAkJCX0NCisgCQl9DQorIAkJYXJnbGVuID0gVkFS
U0laRShzdHJpbmcpIC0gVkFSSERSU1o7DQorIAkJaWYgKGFyZ2xlbiAqIHNp
emVvZihVQ2hhcikgPj0gU1RBQ0tCVUZMRU4pDQorIAkJew0KKyAJCQlidWZs
ZW4gPSBhcmdsZW4gKiBzaXplb2YoVUNoYXIpICsgMTsNCisgIAkJCXNvdXJj
ZSA9IHBhbGxvYyhidWZsZW4pOw0KKyAJCQlkZXN0ID0gcGFsbG9jKGJ1Zmxl
bik7DQorIAkJfQ0KKyAJCWVsc2UNCisgCQl7DQorIAkJCWJ1ZmxlbiA9IFNU
QUNLQlVGTEVOOw0KKyAJCQlzb3VyY2UgPSBzb3VyY2VidWY7DQorIAkJCWRl
c3QgPSBkZXN0YnVmOw0KKyAJCX0NCisgCQkvLyBjb252ZXJ0IHRvIFVURi0x
Ng0KKyAJCXVjbnZfdG9VQ2hhcnMoY29udiwgc291cmNlLCBidWZsZW4sIFZB
UkRBVEEoc3RyaW5nKSwgYXJnbGVuLCAmc3RhdHVzKTsNCisgCQlpZiAoVV9G
QUlMVVJFKHN0YXR1cykpDQorIAkJew0KKyAJCQllcmVwb3J0KEVSUk9SLA0K
KyAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KKyAJCQkJCQkgZXJybXNnKCJJ
Q1UgZXJyb3I6IENvdWxkIG5vdCBjb252ZXJ0IHN0cmluZyIpKSk7DQorIAkJ
fQ0KKyANCisgCQkvLyBydW4gZGVzaXJlZCBmdW5jdGlvbg0KKyAJCWJ1Zmxl
biA9IHVfc3RyVG9UaXRsZShkZXN0LCBidWZsZW4sIHNvdXJjZSwgLTEsIE5V
TEwsIE5VTEwsICZzdGF0dXMpOw0KKyAJCWlmIChVX0ZBSUxVUkUoc3RhdHVz
KSkNCisgCQl7DQorIAkJCWVyZXBvcnQoRVJST1IsDQorIAkJCQkJCShlcnJj
b2RlKHN0YXR1cyksDQorIAkJCQkJCSBlcnJtc2coIklDVSBlcnJvcjogQ291
bGQgbm90IG1vZGlmeSBjYXNlIikpKTsNCisgCQl9DQorIA0KKyAJCS8vIGFu
ZCBjb252ZXJ0IG1vZGlmaWVkIHV0Zi0xNiBzdHJpbmcgYmFjayB0byB0ZXh0
DQorIAkJcmVzdWx0ID0gVUNoYXJ0b3RleHQoZGVzdCwgYnVmbGVuKTsNCisg
DQorIAkJaWYgKGFyZ2xlbiAqIHNpemVvZihVQ2hhcikgPj0gU1RBQ0tCVUZM
RU4pDQorIAkJew0KKyAJCQlwZnJlZShzb3VyY2UpOw0KKyAJCQlwZnJlZShk
ZXN0KTsNCisgCQl9DQorIAkJUEdfUkVUVVJOX1RFWFRfUChyZXN1bHQpOw0K
KyAJfQ0KKyAJZWxzZQ0KKyAjZWxzZQ0KICAjaWZkZWYgVVNFX1dJREVfVVBQ
RVJfTE9XRVINCiAgCS8qDQogIAkgKglVc2Ugd2lkZSBjaGFyIGNvZGUgb25s
eSB3aGVuIG1heCBlbmNvZGluZyBsZW5ndGggPiAxIGFuZCBjdHlwZSAhPSBD
Lg0KKioqKioqKioqKioqKioqDQoqKiogMzM0LDMzOSAqKioqDQotLS0gNTgx
LDU4NyAtLS0tDQogIAl9DQogIAllbHNlDQogICNlbmRpZiAgIC8qIFVTRV9X
SURFX1VQUEVSX0xPV0VSICovDQorICNlbmRpZiAgIC8qIFVTRV9JQ1UgKi8N
CiAgCXsNCiAgCQl0ZXh0CSAgICpzdHJpbmcgPSBQR19HRVRBUkdfVEVYVF9Q
X0NPUFkoMCk7DQogIAkJaW50CQkJd2FzYWxudW0gPSAwOw0KSW5kZXg6IHNy
Yy9iYWNrZW5kL3V0aWxzL2FkdC92YXJsZW5hLmMNCj09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT0NClJDUyBmaWxlOiAvcHJvamVjdHMvY3Zzcm9vdC9wZ3NxbC9z
cmMvYmFja2VuZC91dGlscy9hZHQvdmFybGVuYS5jLHYNCnJldHJpZXZpbmcg
cmV2aXNpb24gMS4xMzANCmRpZmYgLWMgLXIxLjEzMCB2YXJsZW5hLmMNCioq
KiBzcmMvYmFja2VuZC91dGlscy9hZHQvdmFybGVuYS5jCTI5IEp1bCAyMDA1
IDAzOjE3OjU1IC0wMDAwCTEuMTMwDQotLS0gc3JjL2JhY2tlbmQvdXRpbHMv
YWR0L3ZhcmxlbmEuYwkzMCBKdWwgMjAwNSAxNzo0NjozNCAtMDAwMA0KKioq
KioqKioqKioqKioqDQoqKiogMzAsMzUgKioqKg0KLS0tIDMwLDQxIC0tLS0N
CiAgI2luY2x1ZGUgInV0aWxzL3BnX2xvY2FsZS5oIg0KICAjaW5jbHVkZSAi
cmVnZXgvcmVnZXguaCINCiAgDQorICNpZmRlZiBVU0VfSUNVDQorICNpbmNs
dWRlIDx1bmljb2RlL3V0eXBlcy5oPiAgIC8qIEJhc2ljIElDVSBkYXRhIHR5
cGVzICovDQorICNpbmNsdWRlIDx1bmljb2RlL3VjbnYuaD4gICAgIC8qIEMg
ICBDb252ZXJ0ZXIgQVBJICAgICovDQorICNpbmNsdWRlIDx1bmljb2RlL3Vj
b2wuaD4NCisgI2luY2x1ZGUgPHVuaWNvZGUvdWxvYy5oPg0KKyAjZW5kaWYg
LyogVVNFX0lDVSAqLw0KICANCiAgdHlwZWRlZiBzdHJ1Y3QgdmFybGVuYSB1
bmtub3duOw0KICANCioqKioqKioqKioqKioqKg0KKioqIDg0NCw4NzQgKioq
Kg0KICANCiAgCWlmICghbGNfY29sbGF0ZV9pc19jKCkpDQogIAl7DQohIAkJ
Y2hhcgkJYTFidWZbU1RBQ0tCVUZMRU5dOw0KISAJCWNoYXIJCWEyYnVmW1NU
QUNLQlVGTEVOXTsNCiEgCQljaGFyCSAgICphMXAsDQohIAkJCQkgICAqYTJw
Ow0KICANCiEgCQlpZiAobGVuMSA+PSBTVEFDS0JVRkxFTikNCiEgCQkJYTFw
ID0gKGNoYXIgKikgcGFsbG9jKGxlbjEgKyAxKTsNCiEgCQllbHNlDQohIAkJ
CWExcCA9IGExYnVmOw0KISAJCWlmIChsZW4yID49IFNUQUNLQlVGTEVOKQ0K
ISAJCQlhMnAgPSAoY2hhciAqKSBwYWxsb2MobGVuMiArIDEpOw0KICAJCWVs
c2UNCiEgCQkJYTJwID0gYTJidWY7DQogIA0KISAJCW1lbWNweShhMXAsIGFy
ZzEsIGxlbjEpOw0KISAJCWExcFtsZW4xXSA9ICdcMCc7DQohIAkJbWVtY3B5
KGEycCwgYXJnMiwgbGVuMik7DQohIAkJYTJwW2xlbjJdID0gJ1wwJzsNCiEg
DQohIAkJcmVzdWx0ID0gc3RyY29sbChhMXAsIGEycCk7DQohIA0KISAJCWlm
IChsZW4xID49IFNUQUNLQlVGTEVOKQ0KISAJCQlwZnJlZShhMXApOw0KISAJ
CWlmIChsZW4yID49IFNUQUNLQlVGTEVOKQ0KISAJCQlwZnJlZShhMnApOw0K
ICAJfQ0KICAJZWxzZQ0KICAJew0KLS0tIDg1MCw5ODYgLS0tLQ0KICANCiAg
CWlmICghbGNfY29sbGF0ZV9pc19jKCkpDQogIAl7DQohICNpZmRlZiBVU0Vf
SUNVDQohICNkZWZpbmUgVVNUQUNLQlVGTEVOCQlTVEFDS0JVRkxFTiAvIHNp
emVvZihVQ2hhcikNCiEgCQlpZiAocGdfZGF0YWJhc2VfZW5jb2RpbmdfbWF4
X2xlbmd0aCgpID4gMSkNCiEgCQl7DQohIAkJCVVDaGFyCQlhMWJ1ZltVU1RB
Q0tCVUZMRU5dLA0KISAJCSAgICAgICAgIAkJYTJidWZbVVNUQUNLQlVGTEVO
XTsNCiEgCQkJaW50CQkJYTFsZW4gPSBVU1RBQ0tCVUZMRU4sDQohIAkJCSAg
CQkJYTJsZW4gPSBVU1RBQ0tCVUZMRU47CQ0KISAJCQlVQ2hhcgkgICAqYTFw
LA0KISAJCQkgICAgICAgICAgICphMnA7DQohIA0KISAJCQlzdGF0aWMgVUNv
bGxhdG9yICogY29sbGF0b3IgPSBOVUxMOw0KISAJCQlzdGF0aWMgVUNvbnZl
cnRlciAqIGNvbnYgPSBOVUxMOw0KISAJCQlVRXJyb3JDb2RlICBzdGF0dXMg
PSBVX1pFUk9fRVJST1I7DQogIA0KISAJCQlpZiAoY29udiA9PSBOVUxMKQ0K
ISAJCQl7DQohIAkJCQljb252ID0gdWNudl9vcGVuKE5VTEwsICZzdGF0dXMp
Ow0KISAJCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpIHx8IGNvbnYgPT0gTlVM
TCkNCiEgCQkJCXsNCiEgCQkJCQllcmVwb3J0KEVSUk9SLA0KISAJCQkJCQkJ
KGVycmNvZGUoc3RhdHVzKSwNCiEgCQkJCQkJCSBlcnJtc2coIklDVSBlcnJv
cjogdmFybGVuYS5jLCBjb3VsZCBub3QgZ2V0IGNvbnZlcnRlciBmb3IgXCIl
c1wiIiwgdWNudl9nZXREZWZhdWx0TmFtZSgpKSkpOw0KISAJCQkJfQ0KISAJ
CQl9DQohIA0KISAJCQkvKiBXZSBrZWVwIGEgc3RhdGljIGNvbGxhdG9yICJm
b3JldmVyIiwgc2luY2UgaXQgaXMgaGFyZA0KISAJCQkgKiBjb2RlZCBpbnRv
IHRoZSBkYXRhYmFzZSBjbHVzdGVyIGF0IGluaXRkYiB0aW1lDQohIAkJCSAq
IGFueXdheS4gQ3JlYXRlIGl0IGZpcnN0IHRpbWUgd2UgZ2V0IGhlcmUuICov
DQohIAkJCWlmIChjb2xsYXRvciA9PSBOVUxMKQ0KISAJCQl7DQohIAkJCQkv
KiBFeHBlY3QgTENfQ09MTEFURSB0byBiZSBzZXQgdG8gc29tZXRoaW5nIHRo
YXQgSUNVDQohIAkJCQkgKiB3aWxsIHVuZGVyc3RhbmQuIFRoaXMgaXMgcXVp
dGUgcHJvYmFibGUsIHNpbmNlIElDVQ0KISAJCQkJICogZG9lcyBhIGxvdCBv
ZiBoZXVyaXN0aWNzIHdpdGggdGhpcyBhcmd1bWVudC4gSSdkDQohIAkJCQkg
KiByYXRoZXIgc2V0IHRoaXMgaW4geGxvZy5jLCBidXQgaXQgc2VlbXMgSUNV
IGZvcmdldHMNCiEgCQkJCSAqIGl0Pz8/ICovDQohICNpZm5kZWYgV0lOMzIN
CiEgCQkJCXVsb2Nfc2V0RGVmYXVsdChzZXRsb2NhbGUoTENfQ09MTEFURSwg
TlVMTCksICZzdGF0dXMpOw0KISAjZWxzZQ0KISAJCQkJLyogV2luMzIgbG9j
YWxlIG5hbWVzIGFyZSBjb21wbGV0ZWx5IGRpZmZlcmVudCBmcm9tIHdoYXQg
SUNVIGV4cGVjdHMsIHNvIA0KISAJCQkJICAgd2UgbmVlZCB0byBkbyBzb21l
IGNvbnZlcnNpb24gKi8NCiEgCQkJCXVsb2Nfc2V0RGVmYXVsdChwZ3dpbjMy
X2xvY2FsZW1hcChzZXRsb2NhbGUoTENfQ09MTEFURSwgTlVMTCkpLCAmc3Rh
dHVzKTsNCiEgI2VuZGlmDQohIAkJCQlpZihVX0ZBSUxVUkUoc3RhdHVzKSkN
CiEgCQkJCXsNCiEgCQkJCQllcmVwb3J0KFdBUk5JTkcsDQohIAkJCQkJCQko
ZXJyY29kZShzdGF0dXMpLA0KISAJCQkJCQkJIGVycm1zZygiSUNVIEVycm9y
OiB2YXJsZW5hLmMsIGNvdWxkIG5vdCBzZXQgZGVmYXVsdCBsY19jb2xsYXRl
IikpKTsNCiEgCQkJCX0NCiEgCQkJCWNvbGxhdG9yID0gdWNvbF9vcGVuKE5V
TEwsICZzdGF0dXMpOw0KISAJCQkJaWYgKFVfRkFJTFVSRShzdGF0dXMpKQ0K
ISAJCQkJew0KISAJCQkJCWVyZXBvcnQoV0FSTklORywNCiEgCQkJCQkJCShl
cnJjb2RlKHN0YXR1cyksDQohIAkJCQkJCQkgZXJybXNnKCJJQ1UgRXJyb3I6
IHZhcmxlbmEuYywgY291bGQgbm90IG9wZW4gY29sbGF0b3IiKSkpOw0KISAJ
CQkJfQ0KISAJCQl9DQohIAkJCQ0KISAJCQlpZiAobGVuMSA+PSBVU1RBQ0tC
VUZMRU4gLyBzaXplb2YoVUNoYXIpKQ0KISAJCQl7DQohIAkJCQlhMWxlbiA9
IGxlbjEgKiBzaXplb2YoVUNoYXIpICsgMjsNCiEgCQkJCWExcCA9IChVQ2hh
ciAqKSBwYWxsb2MoYTFsZW4pOwkNCiEgCQkJfQ0KISAJCQllbHNlDQohIAkJ
CQlhMXAgPSBhMWJ1ZjsNCiEgDQohIAkJCWlmIChsZW4yID49IFVTVEFDS0JV
RkxFTiAvIHNpemVvZihVQ2hhcikpDQohIAkJCXsNCiEgCQkJCWEybGVuID0g
bGVuMiAqIHNpemVvZihVQ2hhcikgKyAyOw0KISAJCQkJYTJwID0gKFVDaGFy
ICopIHBhbGxvYyhhMmxlbik7DQohIAkJCX0NCiEgCQkJZWxzZQ0KISAJCQkJ
YTJwID0gYTJidWY7DQohIA0KISAJCQl1Y252X3RvVUNoYXJzKGNvbnYsIGEx
cCwgYTFsZW4sIGFyZzEsIGxlbjEsICZzdGF0dXMpOw0KISAJCQlpZihVX0ZB
SUxVUkUoc3RhdHVzKSkNCiEgCQkJew0KISAJCQkJZXJlcG9ydChXQVJOSU5H
LA0KISAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KISAJCQkJCQkgZXJybXNn
KCJJQ1UgRXJyb3I6IHZhcmxlbmEuYywgY291bGQgbm90IGNvbnZlcnQgdG8g
VUNoYXJzIikpKTsNCiEgCQkJfQ0KISAJCQl1Y252X3RvVUNoYXJzKGNvbnYs
IGEycCwgYTJsZW4sIGFyZzIsIGxlbjIsICZzdGF0dXMpOw0KISAJCQlpZihV
X0ZBSUxVUkUoc3RhdHVzKSkNCiEgCQkJew0KISAJCQkJZXJlcG9ydChXQVJO
SU5HLA0KISAJCQkJCQkoZXJyY29kZShzdGF0dXMpLA0KISAJCQkJCQkgZXJy
bXNnKCJJQ1UgRXJyb3I6IHZhcmxlbmEuYywgY291bGQgbm90IGNvbnZlcnQg
dG8gVUNoYXJzIikpKTsNCiEgCQkJfQ0KISANCiEgCQkJcmVzdWx0ID0gdWNv
bF9zdHJjb2xsKGNvbGxhdG9yLCBhMXAsIC0xLCBhMnAsIC0xKTsNCiEgCQkJ
aWYoVV9GQUlMVVJFKHN0YXR1cykpDQohIAkJCXsNCiEgCQkJCWVyZXBvcnQo
V0FSTklORywNCiEgCQkJCQkJKGVycmNvZGUoc3RhdHVzKSwNCiEgCQkJCQkJ
IGVycm1zZygiSUNVIEVycm9yOiB2YXJsZW5hLmMsIGNvdWxkIG5vdCBjb2xs
YXRlIikpKTsNCiEgCQkJfQ0KISANCiEgCQkJaWYgKGxlbjEgKiBzaXplb2Yo
VUNoYXIpID49IFVTVEFDS0JVRkxFTikNCiEgCQkJCXBmcmVlKGExcCk7DQoh
IAkJCWlmIChsZW4yICogc2l6ZW9mKFVDaGFyKSA+PSBVU1RBQ0tCVUZMRU4p
DQohIAkJCQlwZnJlZShhMnApOw0KISAJCX0NCiAgCQllbHNlDQohICNlbmRp
ZiAvKiBVU0VfSUNVICovDQohIAkJew0KISAJCQljaGFyCQlhMWJ1ZltTVEFD
S0JVRkxFTl07DQohIAkJCWNoYXIJCWEyYnVmW1NUQUNLQlVGTEVOXTsNCiEg
CQkJY2hhcgkgICAqYTFwLA0KISAJCQkJCSAgICphMnA7DQogIA0KISAJCQlp
ZiAobGVuMSA+PSBTVEFDS0JVRkxFTikNCiEgCQkJCWExcCA9IChjaGFyICop
IHBhbGxvYyhsZW4xICsgMSk7DQohIAkJCWVsc2UNCiEgCQkJCWExcCA9IGEx
YnVmOw0KISAJCQlpZiAobGVuMiA+PSBTVEFDS0JVRkxFTikNCiEgCQkJCWEy
cCA9IChjaGFyICopIHBhbGxvYyhsZW4yICsgMSk7DQohIAkJCWVsc2UNCiEg
CQkJCWEycCA9IGEyYnVmOw0KISAJCQkNCiEgCQkJbWVtY3B5KGExcCwgYXJn
MSwgbGVuMSk7DQohIAkJCWExcFtsZW4xXSA9ICdcMCc7DQohIAkJCW1lbWNw
eShhMnAsIGFyZzIsIGxlbjIpOw0KISAJCQlhMnBbbGVuMl0gPSAnXDAnOw0K
ISAJCQkNCiEgCQkJcmVzdWx0ID0gc3RyY29sbChhMXAsIGEycCk7DQohIAkJ
CQ0KISAJCQlpZiAobGVuMSA+PSBTVEFDS0JVRkxFTikNCiEgCQkJCXBmcmVl
KGExcCk7DQohIAkJCWlmIChsZW4yID49IFNUQUNLQlVGTEVOKQ0KISAJCQkJ
cGZyZWUoYTJwKTsNCiEgCQl9DQogIAl9DQogIAllbHNlDQogIAl7DQpJbmRl
eDogc3JjL2JhY2tlbmQvdXRpbHMvbWIvZW5jbmFtZXMuYw0KPT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PQ0KUkNTIGZpbGU6IC9wcm9qZWN0cy9jdnNyb290L3Bn
c3FsL3NyYy9iYWNrZW5kL3V0aWxzL21iL2VuY25hbWVzLmMsdg0KcmV0cmll
dmluZyByZXZpc2lvbiAxLjI1DQpkaWZmIC1jIC1yMS4yNSBlbmNuYW1lcy5j
DQoqKiogc3JjL2JhY2tlbmQvdXRpbHMvbWIvZW5jbmFtZXMuYwkxNCBNYXIg
MjAwNSAxODozMToyMCAtMDAwMAkxLjI1DQotLS0gc3JjL2JhY2tlbmQvdXRp
bHMvbWIvZW5jbmFtZXMuYwkyOCBKdWwgMjAwNSAxNzoxNjo1MiAtMDAwMA0K
KioqKioqKioqKioqKioqDQoqKiogMzc1LDM4MCAqKioqDQotLS0gMzc1LDQ5
NiAtLS0tDQogIAl9DQogIH07DQogIA0KKyAjaWZkZWYgVVNFX0lDVQ0KKyAv
Kg0KKyAgKiBUcnkgdG8gbWFwIG1vc3QgaW50ZXJuYWwgY2hhcmFjdGVyIGVu
Y29kaW5ncyB0byB0aGUgcHJvcGVyIGFuZA0KKyAgKiBwcmVmZXJyZWQgSUFO
QSBzdHJpbmcuIFVzZSB0aGlzIGluIG1idXRpbHMuYyB0byBmZWVkIElDVSBp
bmZvIGFib3V0DQorICAqIHRoZSBkYXRhYmFzZSdzIGNoYXJhY3RlciBlbmNv
ZGluZy4NCisgICoNCisgICogUGFsbGUgR2lyZ2Vuc29obiwgMjAwNQ0KKyAg
Ki8NCisgDQorIHBnX2VuYzJuYW1lIHBnX2VuYzJpYW5hbmFtZV90YmxbXSA9
DQorIHsNCisgCXsNCisgCQkiVVMtQVNDSUkiLCBQR19TUUxfQVNDSUkNCisg
CX0sDQorIAl7DQorIAkJIkVVQy1KUCIsIFBHX0VVQ19KUA0KKyAJfSwNCisg
CXsNCisgCQkiR0IyMzEyIiwgUEdfRVVDX0NODQorIAl9LA0KKyAJew0KKyAJ
CSJFVUMtS1IiLCBQR19FVUNfS1INCisgCX0sDQorIAl7DQorIAkJIklTTy0y
MDIyLUNOIiwgUEdfRVVDX1RXDQorIAl9LA0KKyAJew0KKyAJCSJLU19DXzU2
MDEtMTk4NyIsIFBHX0pPSEFCICAvKiBlaXRoZXIgS1NfQ181NjAxLTE5ODcg
b3IgSVNPLTIwMjItS1IgPz8/ICovDQorIAl9LA0KKyAJew0KKyAJCSJVVEYt
OCIsIFBHX1VURjgNCisgCX0sDQorIAl7DQorIAkJIk1VTEVfSU5URVJOQUwi
LCBQR19NVUxFX0lOVEVSTkFMICAvKiBpcyBub3QgZm9yIHJlYWwgKi8NCisg
CX0sDQorIAl7DQorIAkJIklTTy04ODU5LTEiLCBQR19MQVRJTjENCisgCX0s
DQorIAl7DQorIAkJIklTTy04ODU5LTIiLCBQR19MQVRJTjINCisgCX0sDQor
IAl7DQorIAkJIklTTy04ODU5LTMiLCBQR19MQVRJTjMNCisgCX0sDQorIAl7
DQorIAkJIklTTy04ODU5LTQiLCBQR19MQVRJTjQNCisgCX0sDQorIAl7DQor
IAkJIklTTy04ODU5LTkiLCBQR19MQVRJTjUNCisgCX0sDQorIAl7DQorIAkJ
IklTTy04ODU5LTEwIiwgUEdfTEFUSU42DQorIAl9LA0KKyAJew0KKyAJCSJJ
U08tODg1OS0xMyIsIFBHX0xBVElONw0KKyAJfSwNCisgCXsNCisgCQkiSVNP
LTg4NTktMTQiLCBQR19MQVRJTjgNCisgCX0sDQorIAl7DQorIAkJIklTTy04
ODU5LTE1IiwgUEdfTEFUSU45DQorIAl9LA0KKyAJew0KKyAJCSJJU08tODg1
OS0xNiIsIFBHX0xBVElOMTANCisgCX0sDQorIAl7DQorIAkJIndpbmRvd3Mt
MTI1NiIsIFBHX1dJTjEyNTYNCisgCX0sDQorIAl7DQorIAkJIndpbmRvd3Mt
MTI1OCIsIFBHX1dJTjEyNTgNCisgCX0sDQorIAl7DQorIAkJIndpbmRvd3Mt
ODc0IiwgUEdfV0lOODc0DQorIAl9LA0KKyAJew0KKyAJCSJLT0k4LVIiLCBQ
R19LT0k4Ug0KKyAJfSwNCisgCXsNCisgCQkid2luZG93cy0xMjUxIiwgUEdf
V0lOMTI1MQ0KKyAJfSwNCisgCXsNCisgCQkiSUJNODY2IiwgUEdfV0lOODY2
DQorIAl9LA0KKyAJew0KKyAJCSJJU08tODg1OS01IiwgUEdfSVNPXzg4NTlf
NQ0KKyAJfSwNCisgCXsNCisgCQkiSVNPLTg4NTktNiIsIFBHX0lTT184ODU5
XzYNCisgCX0sDQorIAl7DQorIAkJIklTTy04ODU5LTciLCBQR19JU09fODg1
OV83DQorIAl9LA0KKyAJew0KKyAJCSJJU08tODg1OS04IiwgUEdfSVNPXzg4
NTlfOA0KKyAJfSwNCisgCXsNCisgCQkid2luZG93cy0xMjUwIiwgUEdfV0lO
MTI1MA0KKyAJfSwNCisgCXsNCisgCQkiU2hpZnRfSklTIiwgUEdfU0pJUw0K
KyAJfSwNCisgCXsNCisgCQkiQmlnNSIsIFBHX0JJRzUNCisgCX0sDQorIAl7
DQorIAkJIkdCSyIsIFBHX0dCSw0KKyAJfSwNCisgCXsNCisgCQkiY3A5NDki
LCBQR19VSEMNCisgCX0sDQorIAl7DQorIAkJIkdCMTgwMzAiLCBQR19HQjE4
MDMwDQorIAl9DQorIH07DQorICNlbmRpZiAvKiBVU0VfSUNVICovDQorIA0K
ICAvKiAtLS0tLS0tLS0tDQogICAqIEVuY29kaW5nIGNoZWNrcywgZm9yIGVy
cm9yIHJldHVybnMgLTEgZWxzZSBlbmNvZGluZyBpZA0KICAgKiAtLS0tLS0t
LS0tDQpJbmRleDogc3JjL2JhY2tlbmQvdXRpbHMvbWIvbWJ1dGlscy5jDQo9
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09DQpSQ1MgZmlsZTogL3Byb2plY3RzL2N2
c3Jvb3QvcGdzcWwvc3JjL2JhY2tlbmQvdXRpbHMvbWIvbWJ1dGlscy5jLHYN
CnJldHJpZXZpbmcgcmV2aXNpb24gMS41MA0KZGlmZiAtYyAtcjEuNTAgbWJ1
dGlscy5jDQoqKiogc3JjL2JhY2tlbmQvdXRpbHMvbWIvbWJ1dGlscy5jCTEw
IEp1bCAyMDA1IDIxOjEzOjU5IC0wMDAwCTEuNTANCi0tLSBzcmMvYmFja2Vu
ZC91dGlscy9tYi9tYnV0aWxzLmMJMjggSnVsIDIwMDUgMTY6MzY6NTQgLTAw
MDANCioqKioqKioqKioqKioqKg0KKioqIDE1LDIwICoqKioNCi0tLSAxNSwy
MyAtLS0tDQogICNpbmNsdWRlICJ1dGlscy9tZW11dGlscy5oIg0KICAjaW5j
bHVkZSAidXRpbHMvc3lzY2FjaGUuaCINCiAgI2luY2x1ZGUgImNhdGFsb2cv
bmFtZXNwYWNlLmgiDQorICNpZmRlZiBVU0VfSUNVDQorICNpbmNsdWRlIDx1
bmljb2RlL3VjbnYuaD4NCisgI2VuZGlmIC8qIFVTRV9JQ1UgKi8NCiAgDQog
IC8qDQogICAqIFdlIGhhbmRsZSBmb3IgYWN0dWFsIEZFIGFuZCBCRSBlbmNv
ZGluZyBzZXR0aW5nIGVuY29kaW5nLWlkZW50aWZpY2F0b3INCioqKioqKioq
KioqKioqKg0KKioqIDU3Niw1ODEgKioqKg0KLS0tIDU3OSw1ODcgLS0tLQ0K
ICANCiAgCURhdGFiYXNlRW5jb2RpbmcgPSAmcGdfZW5jMm5hbWVfdGJsW2Vu
Y29kaW5nXTsNCiAgCUFzc2VydChEYXRhYmFzZUVuY29kaW5nLT5lbmNvZGlu
ZyA9PSBlbmNvZGluZyk7DQorICNpZmRlZiBVU0VfSUNVDQorIAl1Y252X3Nl
dERlZmF1bHROYW1lKCgmcGdfZW5jMmlhbmFuYW1lX3RibFtlbmNvZGluZ10p
LT5uYW1lKTsNCisgI2VuZGlmDQogIH0NCiAgDQogIHZvaWQNCkluZGV4OiBz
cmMvaW5jbHVkZS9wZ19jb25maWcuaC5pbg0KPT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PQ0KUkNTIGZpbGU6IC9wcm9qZWN0cy9jdnNyb290L3Bnc3FsL3NyYy9p
bmNsdWRlL3BnX2NvbmZpZy5oLmluLHYNCnJldHJpZXZpbmcgcmV2aXNpb24g
MS44Ng0KZGlmZiAtYyAtcjEuODYgcGdfY29uZmlnLmguaW4NCioqKiBzcmMv
aW5jbHVkZS9wZ19jb25maWcuaC5pbgkxIEp1bCAyMDA1IDE4OjE3OjMxIC0w
MDAwCTEuODYNCi0tLSBzcmMvaW5jbHVkZS9wZ19jb25maWcuaC5pbgkyOCBK
dWwgMjAwNSAxNjozNjo1NCAtMDAwMA0KKioqKioqKioqKioqKioqDQoqKiog
NjQyLDY0NyAqKioqDQotLS0gNjQyLDY1MCAtLS0tDQogIC8qIERlZmluZSB0
byBidWlsZCB3aXRoIChPcGVuKVNTTCBzdXBwb3J0LiAoLS13aXRoLW9wZW5z
c2wpICovDQogICN1bmRlZiBVU0VfU1NMDQogIA0KKyAvKiBEZWZpbmUgdG8g
YnVpbGQgd2l0aCBJQ1Ugc3VwcG9ydC4gKC0td2l0aC1pY3UpICovDQorICN1
bmRlZiBVU0VfSUNVDQorIA0KICAvKiBEZWZpbmUgdG8gc2VsZWN0IFN5c1Yt
c3R5bGUgc2VtYXBob3Jlcy4gKi8NCiAgI3VuZGVmIFVTRV9TWVNWX1NFTUFQ
SE9SRVMNCiAgDQpJbmRleDogc3JjL2luY2x1ZGUvbWIvcGdfd2NoYXIuaA0K
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PQ0KUkNTIGZpbGU6IC9wcm9qZWN0cy9j
dnNyb290L3Bnc3FsL3NyYy9pbmNsdWRlL21iL3BnX3djaGFyLmgsdg0KcmV0
cmlldmluZyByZXZpc2lvbiAxLjU5DQpkaWZmIC1jIC1yMS41OSBwZ193Y2hh
ci5oDQoqKiogc3JjL2luY2x1ZGUvbWIvcGdfd2NoYXIuaAkxNSBKdW4gMjAw
NSAwMDoxNTowOCAtMDAwMAkxLjU5DQotLS0gc3JjL2luY2x1ZGUvbWIvcGdf
d2NoYXIuaAkyOCBKdWwgMjAwNSAxNjozNjo1NCAtMDAwMA0KKioqKioqKioq
KioqKioqDQoqKiogMjM1LDI0MCAqKioqDQotLS0gMjM1LDI0MSAtLS0tDQog
IH0gcGdfZW5jMm5hbWU7DQogIA0KICBleHRlcm4gcGdfZW5jMm5hbWUgcGdf
ZW5jMm5hbWVfdGJsW107DQorIGV4dGVybiBwZ19lbmMybmFtZSBwZ19lbmMy
aWFuYW5hbWVfdGJsW107DQogIA0KICBleHRlcm4gcGdfZW5jbmFtZSAqcGdf
Y2hhcl90b19lbmNuYW1lX3N0cnVjdChjb25zdCBjaGFyICpuYW1lKTsNCiAg
DQpJbmRleDogc3JjL2luY2x1ZGUvcG9ydC93aW4zMi5oDQo9PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09DQpSQ1MgZmlsZTogL3Byb2plY3RzL2N2c3Jvb3QvcGdz
cWwvc3JjL2luY2x1ZGUvcG9ydC93aW4zMi5oLHYNCnJldHJpZXZpbmcgcmV2
aXNpb24gMS40Ng0KZGlmZiAtYyAtcjEuNDYgd2luMzIuaA0KKioqIHNyYy9p
bmNsdWRlL3BvcnQvd2luMzIuaAkxNiBKdW4gMjAwNSAxNzo1Mzo1NCAtMDAw
MAkxLjQ2DQotLS0gc3JjL2luY2x1ZGUvcG9ydC93aW4zMi5oCTMwIEp1bCAy
MDA1IDE3OjM5OjM2IC0wMDAwDQoqKioqKioqKioqKioqKioNCioqKiAyNTYs
MjU4ICoqKioNCi0tLSAyNTYsMjYxIC0tLS0NCiAgDQogIC8qIGluIGJhY2tl
bmQvcG9ydC93aW4zMi9lcnJvci5jICovDQogIGV4dGVybiB2b2lkIF9kb3Nt
YXBlcnIodW5zaWduZWQgbG9uZyk7DQorIA0KKyAvKiBpbiBiYWNrZW5kL3Bv
cnQvd2luMzIvbG9jYWxlbWFwLmMgKi8NCisgZXh0ZXJuIGNoYXIgKnBnd2lu
MzJfbG9jYWxlbWFwKGNoYXIgKndpbmxvY2FsZSk7DQo=

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="localemap.pl"
Content-Transfer-Encoding: base64
Content-Description: localemap.pl
Content-Disposition: attachment;
	filename="localemap.pl"

IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMNCiMgbG9jYWxlbWFwLnBs
IC0tIEJ1aWxkIGxvY2FsZSBtYXAgYmFzZWQgb24gc291cmNlIGRhdGEgZmls
ZXMNCiMNCiMgQ29weXJpZ2h0IChjKSAyMDA1LCBQb3N0Z3JlU1FMIEdsb2Jh
bCBEZXZlbG9wbWVudCBHcm91cA0KIw0KIyAkUG9zdGdyZVNRTCQNCiMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjDQojDQojIFRoZSBmaWxlIGlzbzYz
OSBjb250YWlucyBJU08tNjM5IGxhbmd1YWdlIGNvZGVzLCBjdXQtYW5kLXBh
c3RlIGZyb20NCiMgaHR0cDovL3d3dy53My5vcmcvV0FJL0VSL0lHL2VydC9p
c282MzkuaHRtLg0KIw0KIyBUaGUgZmlsZSBpc28zMTY2IGNvbnRhaW5zIElT
Ty0zMTY2IGNvdW50cnkgY29kZXMsIGN1dC1hbmQtcGFzdGUgZnJvbQ0KIyBo
dHRwOi8vd3d3Lm9hc2lzLW9wZW4ub3JnL2NvdmVyL2NvdW50cnkzMTY2Lmh0
bWwsDQojDQoNCm9wZW4oSSwiaXNvNjM5IikgfHwgZGllICJDb3VsZCBub3Qg
b3BlbiBpc282MzlcbiI7DQp3aGlsZSAoPEk+KSB7DQoJY2hvbXA7DQoJZGll
ICJNYWxmb3JtYXR0ZWQgbGluZSBpbiBpc282MzkiIHVubGVzcyAvXihbQS1a
XXsyfSkgXCIoLispXCIkLzsNCgkkY29kZSA9IGxjICQxOw0KCSRsYW5nID0g
bGMgJDI7DQoJaWYgKCRsYW5nID1+IC9eKC4rKVwiIC8pIHsgJGxhbmcgPSAk
MTsgfQ0KCSRpc282Mzl7JGxhbmd9PSRjb2RlOw0KfQ0KY2xvc2UoSSk7DQoN
Cm9wZW4oSSwiaXNvMzE2NiIpIHx8IGRpZSAiQ291bGQgbm90IG9wZW4gaXNv
MzE2NlxuIjsNCndoaWxlICg8ST4pIHsNCgljaG9tcDsNCglkaWUgIk1hbGZv
cm1hdHRlZCBsaW5lIGluIGlzbzMxNjYiIHVubGVzcyAvXihbQS1aXXsyfSlc
cysoLispJC87DQoJJGNvZGUgPSBsYyAkMTsNCgkkY291bnRyeSA9IGxjICQy
Ow0KCSRpc28zMTY2eyRjb3VudHJ5fSA9ICRjb2RlOw0KfQ0KY2xvc2UoSSk7
DQoNCnByaW50IDw8RU9GOw0KdHlwZWRlZiBzdHJ1Y3Qgew0KCWNoYXIgKmtl
eTsNCgljaGFyICp2YWw7DQp9IF9rZXl2YWw7DQoNCnN0YXRpYyBfa2V5dmFs
IGlzbzYzOVtdID0gew0KRU9GDQpmb3IgJGsgKHNvcnQga2V5cyAlaXNvNjM5
KSB7DQoJcHJpbnQgIiB7XCIka1wiLFwiJGlzbzYzOXska31cIn0sXG4iOw0K
fQ0KcHJpbnQgIiB7TlVMTCxOVUxMfVxufTtcblxuIjsNCg0KcHJpbnQgInN0
YXRpYyBfa2V5dmFsIGlzbzMxNjZbXSA9IHtcbiI7DQpmb3IgJGsgKHNvcnQg
a2V5cyAlaXNvMzE2Nikgew0KCXByaW50ICIge1wiJGtcIixcIiRpc28zMTY2
eyRrfVwifSxcbiI7DQp9DQoNCnByaW50ICIge05VTEwsTlVMTH1cbn07XG5c
biI7DQo=

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="localemap.c"
Content-Transfer-Encoding: base64
Content-Description: localemap.c
Content-Disposition: attachment;
	filename="localemap.c"

LyoNCiAqIGxvY2FsZW1hcCAtIG1hcCB3aW4zMiBsb2NhbGUgbmFtZXMgdG8g
SUNVIHN0YW5kYXJkIG5hbWVzDQogKg0KICogQ29weXJpZ2h0IChDKSAyMDA1
LCBQb3N0Z3JlU1FMIEdsb2JhbCBEZXZlbG9wbWVudCBHcm91cA0KICoNCiAq
ICRQb3N0Z3JlU1FMJA0KICovDQojaW5jbHVkZSAicG9zdGdyZXMuaCIgDQoN
CiNpbmNsdWRlICJsb2NhbGVtYXAuaCINCg0KDQpzdGF0aWMgY2hhciAqbWF0
Y2hfbGFuZ3VhZ2UoY2hhciAqd2lubGFuZ3VhZ2UpDQp7DQoJX2tleXZhbCAq
a3Y7DQoNCglmb3IgKGt2ID0gaXNvNjM5OyBrdi0+a2V5OyBrdisrKSB7DQoJ
CWlmICghc3RyY2FzZWNtcChrdi0+a2V5LCB3aW5sYW5ndWFnZSkpDQoJCQly
ZXR1cm4ga3YtPnZhbDsNCgl9DQoJZXJlcG9ydChFUlJPUiwNCgkJCShlcnJt
c2coImNvdWxkIG5vdCBtYXRjaCBsYW5ndWFnZSBmb3IgJXMiLCB3aW5sYW5n
dWFnZSkpKTsNCglyZXR1cm4gTlVMTDsNCn0NCg0Kc3RhdGljIGNoYXIgKm1h
dGNoX2NvdW50cnkoY2hhciAqd2luY291bnRyeSkNCnsNCglfa2V5dmFsICpr
djsNCg0KCWZvciAoa3YgPSBpc28zMTY2OyBrdi0+a2V5OyBrdisrKSB7DQoJ
CWlmICghc3RyY2FzZWNtcChrdi0+a2V5LCB3aW5jb3VudHJ5KSkNCgkJCXJl
dHVybiBrdi0+dmFsOw0KCX0NCglyZXR1cm4gTlVMTDsNCn0NCg0KLyoNCiAq
IEF0dGVtcHQgdG8gY29udmVydCBhIHdpbjMyIGxvY2FsZSAoU3dlZGlzaF9T
d2VkZW4uMTI1MikNCiAqIHRvIElDVSBmb3JtYXQgKHN2X3NlKQ0KICovDQpz
dGF0aWMgY2hhciBvdXRidWZbMTZdOw0KY2hhciAqcGd3aW4zMl9sb2NhbGVt
YXAoY2hhciAqd2lubG9jYWxlKQ0Kew0KCWNoYXIgYnVmWzI1Nl07DQoJY2hh
ciAqdW5kZXJzY29yZTsNCgljaGFyICpkb3Q7DQoJY2hhciAqY291bnRyeTsN
Cg0KCWlmIChzdHJsZW4od2lubG9jYWxlKT4yNTUpIA0KCQllcmVwb3J0KEVS
Uk9SLA0KCQkJCShlcnJtc2dfaW50ZXJuYWwoImxvY2FsZSBuYW1lIHRvbyBs
b25nIikpKTsNCglzdHJjcHkoYnVmLHdpbmxvY2FsZSk7DQoNCgl1bmRlcnNj
b3JlID0gc3RyY2hyKGJ1ZiwnXycpOw0KCWlmICghdW5kZXJzY29yZSkNCgkJ
LyogT25seSBsYW5ndWFnZSBuYW1lICovDQoJCXJldHVybiBtYXRjaF9sYW5n
dWFnZShidWYpOw0KDQoJKnVuZGVyc2NvcmUgPSAwOw0KCXVuZGVyc2NvcmUr
KzsNCg0KCWRvdCA9IHN0cmNocih1bmRlcnNjb3JlLCcuJyk7DQoJaWYgKGRv
dCkNCgkJLyogSWYgY29kZXBhZ2UgaXMgaW5jbHVkZWQsIHdlIGp1c3QgaWdu
b3JlIGl0LiAqLw0KCQkqZG90ID0gMDsNCgkNCgljb3VudHJ5ID0gbWF0Y2hf
Y291bnRyeSh1bmRlcnNjb3JlKTsNCglpZiAoIWNvdW50cnkpDQoJCXJldHVy
biBtYXRjaF9sYW5ndWFnZShidWYpOw0KDQoJc3ByaW50ZihvdXRidWYsIiVz
XyVzIixtYXRjaF9sYW5ndWFnZShidWYpLGNvdW50cnkpOw0KCXJldHVybiBv
dXRidWY7DQp9DQo=

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

               http://archives.postgresql.org

------_=_NextPart_001_01C5993C.1E1CB100--

From pgsql-hackers-owner+M72011@postgresql.org Wed Aug 24 15:37:17 2005
Return-path: <pgsql-hackers-owner+M72011@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j7OJbH103190
	for <pgman@candle.pha.pa.us>; Wed, 24 Aug 2005 15:37:17 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id D5152B21F6B;
	Wed, 24 Aug 2005 19:37:12 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 34C03D7357
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Wed, 24 Aug 2005 15:53:55 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 85477-03
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Wed, 24 Aug 2005 18:53:52 +0000 (GMT)
Received: from rodrick.geeknet.com.au (ns1.geeknet.com.au [220.244.63.182])
	by svr1.postgresql.org (Postfix) with ESMTP id 7C774D78F9
	for <pgsql-hackers@postgresql.org>; Wed, 24 Aug 2005 15:53:48 -0300 (ADT)
Subject: Re: [HACKERS] FreeBSD ICU  was Win32 unicode vs ICU 
MIME-Version: 1.0
Date: Thu, 25 Aug 2005 04:53:47 +1000
Content-Type: text/plain;
	charset="us-ascii"
Message-ID: <5066E5A966339E42AA04BA10BA706AE50A938F@rodrick.geeknet.com.au>
content-class: urn:content-classes:message
X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0
Thread-Topic: [HACKERS] FreeBSD ICU  was Win32 unicode vs ICU 
Thread-Index: AcWo0l6W575cyph2Twi7Fhfaa+w/tAACptMA
From: "John Hansen" <john@geeknet.com.au>
To: "Kevin McArthur" <postgresql-list@stormtide.ca>,
   <pgsql-hackers@postgresql.org>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.017 required=5 tests=[AWL=0.017]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by candle.pha.pa.us id j7OJbH103190
Status: OR

Kevin McArthur Wrote:

> Should the postgresql project also be looking at CLDR for 
> cross-platform unicode support?

Afaict, from the ICU website, ICU too uses CLDR.
Why reinvent the wheel?

... John

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72581@postgresql.org Sat Sep  3 16:47:42 2005
Return-path: <pgsql-hackers-owner+M72581@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Klg129281
	for <pgman@candle.pha.pa.us>; Sat, 3 Sep 2005 16:47:42 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id CA484B21F00;
	Sat,  3 Sep 2005 20:47:36 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 354A7D8B35
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sat,  3 Sep 2005 17:35:06 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 55971-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sat,  3 Sep 2005 20:34:58 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 4AFECD8AE7
	for <pgsql-hackers@postgresql.org>; Sat,  3 Sep 2005 17:34:56 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1EBeim-0002xI-00; Sun, 04 Sep 2005 06:34:40 +1000
Date: Sat, 3 Sep 2005 22:34:40 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Greg Stark <gsstark@mit.edu>, pgsql-hackers@postgresql.org
Subject: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
Message-ID: <20050903203434.GA4281@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="mP3DRpeJDSE+ciuQ"
Content-Disposition: inline
In-Reply-To: <14696.1125675741@sss.pgh.pa.us>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--mP3DRpeJDSE+ciuQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Sep 02, 2005 at 11:42:21AM -0400, Tom Lane wrote:
> The objection is fundamentally that a platform-specific implementation
> cannot be our long-term goal, and so expending effort on creating one
> seems like a diversion.  If there were a plan put forward showing how
> this is just a useful way-station, and we could see how we'd later get
> rid of the glibc dependency without throwing away the work already done,
> then it would be a different story.

Well, my patch showed that useful locale work can be acheived with
precisely two functions: newlocale and strxfrm_l.

I'm going to talk about two things: one, the code from Apple. Two, how
we present locale support to users.
---
Now, it would be really nice to take Apple's implementation in Darwin
and use that. What I don't understand is the licence of the code in
Darwin. My interpretation is that stuff in:

http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/

is Apple stuff under APSL, useless to us. And that stuff in:

http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/FreeBSD/

are just patches to FreeBSD and this under the normal BSD license (no
big header claiming the licence change). The good news is that the
majority of what we need is in patch form. The bad news is that the hub
of the good stuff (newlocale, duplocale, freelocale) is under a big fat
APSL licence.

Does anyone know if this code can be used at all by BSD projects or did
they blanket relicence everything?
---
Now, I want to bring up some points relating to including a locale
library in PostgreSQL. Given that none of the BSDs seem really
interested in fixing the issue we'll have to do it ourselves (I don't
see anyone else doing it). We can save ourselves effort by basing it on
FreeBSDs locale code, because then we can use their datafiles, which we
*definitly* don't want to maintain ourselves. Now:

1. FreeBSDs locale list is short, some 48 compared with glibc's 217.
Hopefully Apple can expand on that in a way we can use. But given the
difference we should probably give people a way of falling back to the
system libraries in case there's a locale we don't support.

On the other hand, lots of locales are similar so maybe people can find
ones close enough to work. No, glibc and FreeBSD use different file
formats, so you can't copy them.

Do we want this locale data just for collation, or do we want to be
able to use it for formatting monetary amounts too? This is even more
info to store. Lots of languages use ISO/IEC 14651 for order.

2. Locale data needs to be combined with a charset and compiled to work
with the library. PostgreSQL supports at least 15 charsets but we don't
want to ship compiled versions of all of these (Debian learnt that the
hard way). So, how do we generate the files people need.

  a. Auto-compile on demand. First time a locale is referenced spawn
the compiler to create the locale, then continue. (Ugh)
  b. Add a CREATE LOCALE english AS 'en_US' WITH CHARSET 'utf8'. Then
require the COLLATE clause to refer to this identifier. This has some
appeal, seperating the system names from the PostgreSQL names. It also
gives some info regarding charsets.
  c. Should users be allowed to define new locales?
  d. Should admins be required to create the external files using a
program, say pg_createlocale.

Remember, if you use a latin1 locale to sort utf8 you'll get the wrong
result, so we want to avoid that.

3. Compiled locale files are large. One UTF-8 locale datafile can
exceed a megabyte. Do we want the option of disabling it for small
systems?

4. Do we want the option of running system locale in parallel with the
internal ones?

5. I think we're going to have to deal with the very real possibility
that our locale database will not be as good as some of the system
provided ones. The question is how. This is quite unlike timezones
which are quite standardized and rarely change. That database is quite
well maintained.

Would people object to a configure option that selected:
  --with-locales=3Dinternal     (use pg database)
  --with-locales=3Dsystem       (use system database for win32, glibc or Ma=
cOS X)
  --with-locales=3Dnone         (what we support now, which is neither)

I don't think it will be much of an issue to support this, all the
functions take the same parameters and have almost the same names.

6. Locales for SQL_ASCII. Seems to me you have two options, either
reject COLLATE altogether unless they specify a charset, or don't care
and let the user shoot themselves in the foot if they wish...

BTW, this MacOS locale supports seems to be new for 10.4.2 according to
the CVS log info, can anyone confirm this?

Anyway, I hope this post didn't bore too much. Locale support has been
one of those things that has bugged me for a long time and it would be
nice if there could be some real movement.

Have a nice weekend,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--mP3DRpeJDSE+ciuQ
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDGgjYIB7bNG8LQkwRAr6OAJ9uqNDKDQKWUAY4KiPAazHJ1TsVWwCeJ7sq
7hcILjdgZTQ4LjyPAhWnJwQ=
=5MT9
-----END PGP SIGNATURE-----

--mP3DRpeJDSE+ciuQ--

From pgsql-hackers-owner+M72583@postgresql.org Sat Sep  3 17:54:58 2005
Return-path: <pgsql-hackers-owner+M72583@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Lsv108133
	for <pgman@candle.pha.pa.us>; Sat, 3 Sep 2005 17:54:57 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 3325AB21F00;
	Sat,  3 Sep 2005 21:54:51 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id DA9AED8B14
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sat,  3 Sep 2005 18:44:52 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 88603-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sat,  3 Sep 2005 21:44:51 +0000 (GMT)
Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227])
	by svr1.postgresql.org (Postfix) with ESMTP id 01864D8B50
	for <pgsql-hackers@postgresql.org>; Sat,  3 Sep 2005 18:44:50 -0300 (ADT)
Received: from localhost ([127.0.0.1] helo=stark.xeocode.com)
	by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian))
	id 1EBfog-0004U8-00; Sat, 03 Sep 2005 17:44:50 -0400
To: Martijn van Oosterhout <kleptog@svana.org>
cc: Tom Lane <tgl@sss.pgh.pa.us>, Greg Stark <gsstark@mit.edu>,
   pgsql-hackers@postgresql.org
Subject: Re: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
References: <20050902130420.GA15466@svana.org>
	<87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us>
	<20050903203434.GA4281@svana.org>
In-Reply-To: <20050903203434.GA4281@svana.org>
From: Greg Stark <gsstark@mit.edu>
Organization: The Emacs Conspiracy; member since 1992
Date: 03 Sep 2005 17:44:50 -0400
Message-ID: <87oe79ygfh.fsf@stark.xeocode.com>
Lines: 39
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


Martijn van Oosterhout <kleptog@svana.org> writes:

> 2. Locale data needs to be combined with a charset and compiled to work
> with the library. PostgreSQL supports at least 15 charsets but we don't
> want to ship compiled versions of all of these (Debian learnt that the
> hard way). So, how do we generate the files people need.

That's just one of many lessons learned the hard way by distributions. Nor
will it be the last innovation in this area.

I really find this instinct of wanting to reimplement large swaths of the OS
inside Postgres (and annoying detail-ridden swaths that are hard to get right
and continually evolving too) to be a bad idea.

I can't believe it's harder to maintain an 

#ifdef HAVE_STRCOL_L
#else
#endif

than it is to try to maintain an entire independent locale library. Nor is it
simpler for sysadmins to have to maintain an entirely separate set of locales
independently from the system locales.

If you really are unhappy enough with OS setlocale implementations to want to
try to do this then it would be more helpful to do it outside of Postgres.
Package up the Apple setlocale library as a separate package that anyone can
install on Solaris, BSD, Linux or whatever. Then Postgres can just say "it
works fine with your OS library but your OS library might be very slow. Here's
a third-party library that you can install that is fast and may relieve any
problems you have with collation performance."

But I think that's getting ahead of things. Until Postgres even supports
collations using the OS libraries you won't even know if that's even
necessary.

-- 
greg


---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

               http://archives.postgresql.org

From pgsql-hackers-owner+M72585@postgresql.org Sun Sep  4 08:44:01 2005
Return-path: <pgsql-hackers-owner+M72585@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84Ci0127486
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 08:44:01 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 8ED1DB21F00;
	Sun,  4 Sep 2005 12:43:55 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id D87D4D6E38
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 09:31:31 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 80362-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 12:31:29 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id C0F8DD6D8D
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 09:31:24 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1EBteM-0005wf-00; Sun, 04 Sep 2005 22:31:06 +1000
Date: Sun, 4 Sep 2005 14:31:05 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Greg Stark <gsstark@mit.edu>
cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@postgresql.org
Subject: Re: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
Message-ID: <20050904123103.GA21198@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <87oe79ygfh.fsf@stark.xeocode.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="EeQfGwPcQSOJBaQU"
Content-Disposition: inline
In-Reply-To: <87oe79ygfh.fsf@stark.xeocode.com>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--EeQfGwPcQSOJBaQU
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Sep 03, 2005 at 05:44:50PM -0400, Greg Stark wrote:
> [...] Nor is it
> simpler for sysadmins to have to maintain an entirely separate set of loc=
ales
> independently from the system locales.

Indeed, I was already coming up with mechanisms to determine what
locales the system uses and try to autogenerate them. I agree though,
it's not useful for systems that already have complete locale support.
Why add to the burden?

Anyway, my reading of the specs says that we must support the syntax.
It doesn't say we need to support any orderings other than the default
(ie what we do now).

> If you really are unhappy enough with OS setlocale implementations to wan=
t to
> try to do this then it would be more helpful to do it outside of Postgres.
> Package up the Apple setlocale library as a separate package that anyone =
can
> install on Solaris, BSD, Linux or whatever. Then Postgres can just say "it
> works fine with your OS library but your OS library might be very slow. H=
ere's
> a third-party library that you can install that is fast and may relieve a=
ny
> problems you have with collation performance."

That's why I asked about the patches and files that Apple wrote. What
are the licence restrictions? Would we be able to download the, what,
20 files and distribute it as a library. Being APSL we couldn't include
it in the tarball, but it could be a pgfoundry project or something.

If somebody knows a reason why this could not be done, speak up now because
my reading of the APSL licence tells me it's fine.

> But I think that's getting ahead of things. Until Postgres even supports
> collations using the OS libraries you won't even know if that's even
> necessary.

Well, I added COLLATE support for ORDER BY and CREATE INDEX and it
worked in under 200 lines. I'm thinking ahead and I don't think the
COLLATE rules are that hard. Implementing them seems a bit fiddly. It
may be easiest to consider COLLATE a non-associative operator.

I'm still unsure if I should turn the string comparison operators into
three-argument functions.

Anyway, I'll look into the library issue first.
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--EeQfGwPcQSOJBaQU
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDGukEIB7bNG8LQkwRAkeaAJ9rj5Tz7WPZpp+wWYWjTdjR68o8DwCfZzcZ
uB57noHmcnvefVoaw27bQ5Q=
=nLio
-----END PGP SIGNATURE-----

--EeQfGwPcQSOJBaQU--

From pgsql-hackers-owner+M72582@postgresql.org Sat Sep  3 17:46:57 2005
Return-path: <pgsql-hackers-owner+M72582@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Lkr106932
	for <pgman@candle.pha.pa.us>; Sat, 3 Sep 2005 17:46:57 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 6E192B21F00;
	Sat,  3 Sep 2005 21:46:47 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 7CF09D72B7
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sat,  3 Sep 2005 18:36:14 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 77080-09
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sat,  3 Sep 2005 21:36:13 +0000 (GMT)
Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227])
	by svr1.postgresql.org (Postfix) with ESMTP id 85FC0D6FEB
	for <pgsql-hackers@postgresql.org>; Sat,  3 Sep 2005 18:36:09 -0300 (ADT)
Received: from localhost ([127.0.0.1] helo=stark.xeocode.com)
	by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian))
	id 1EBfgA-0004RS-00; Sat, 03 Sep 2005 17:36:02 -0400
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Greg Stark <gsstark@mit.edu>, Martijn van Oosterhout <kleptog@svana.org>,
   pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
References: <20050902130420.GA15466@svana.org>
	<87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us>
In-Reply-To: <14696.1125675741@sss.pgh.pa.us>
From: Greg Stark <gsstark@mit.edu>
Organization: The Emacs Conspiracy; member since 1992
Date: 03 Sep 2005 17:36:02 -0400
Message-ID: <87u0h1ygu5.fsf@stark.xeocode.com>
Lines: 36
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Tom Lane <tgl@sss.pgh.pa.us> writes:

> Greg Stark <gsstark@mit.edu> writes:
> > I still doesn't get where the hostility towards this functionality comes from.
> 
> We're not really willing to say "here is a piece of syntax REQUIRED
> BY THE SQL SPEC which we only support on some platforms".  readline,
> O_DIRECT, and the like are a completely inappropriate analogy, because
> those are inherently platform-dependent (and not in the spec).

But that's not the case at all. The syntax can be supported everywhere it
would just be somewhat faster on some platforms than others. It's already
reasonably fast on any platform that caches locale information which includes
glibc and presumably other free software libcs. It would be slightly faster if
there are _l functions. And much slower if the libc setlocale implementation
is braindead. But there's nothing wrong with saying "it's slow because your
libc is slow. Compile with this freely available library which has a better
implementation". The programming syntax would still be exactly 100% the same.

> The objection is fundamentally that a platform-specific implementation
> cannot be our long-term goal, and so expending effort on creating one
> seems like a diversion.  If there were a plan put forward showing how
> this is just a useful way-station, and we could see how we'd later get
> rid of the glibc dependency without throwing away the work already done,
> then it would be a different story.

It's not like the actual calls to setlocale are going to be much code. One day
presumably some variant of these _l functions will become entirely standard.
In which case you're talking about potentially "throwing away" 50 lines of
code. The bulk of the code is going to be parsing and implementing the actual
syntax and behaviour of the SQL spec. And in any case I wouldn't expect it to
ever get thrown away. There will be people compiling on RH9 or similar vintage
systems for a long time.

-- 
greg


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq

From pgsql-hackers-owner+M72589@postgresql.org Sun Sep  4 13:20:26 2005
Return-path: <pgsql-hackers-owner+M72589@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84HKQ121839
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 13:20:26 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id E4A85B21FBB;
	Sun,  4 Sep 2005 17:20:23 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id C4CA7D7AEA
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 14:07:02 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 63550-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 17:07:01 +0000 (GMT)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20])
	by svr1.postgresql.org (Postfix) with SMTP id 02A4FD7A6F
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 14:06:59 -0300 (ADT)
Received: (qmail invoked by alias); 04 Sep 2005 17:06:59 -0000
Received: from dsl-082-083-230-040.arcor-ip.net (EHLO colt.pezone.net) [82.83.230.40]
  by mail.gmx.net (mp033) with SMTP; 04 Sep 2005 19:06:59 +0200
X-Authenticated: #495269
From: Peter Eisentraut <peter_e@gmx.net>
To: pgsql-hackers@postgresql.org, Martijn van Oosterhout <kleptog@svana.org>
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
Date: Sun, 4 Sep 2005 19:06:57 +0200
User-Agent: KMail/1.8.1
References: <20050902130420.GA15466@svana.org>
In-Reply-To: <20050902130420.GA15466@svana.org>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-ID: <200509041906.57721.peter_e@gmx.net>
X-Y-GMX-Trusted: 0
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.018 required=5 tests=[AWL=0.018]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Martijn van Oosterhout wrote:
> This is just a proof of concept patch. I didn't send it to -patches
> because as Tom pointed out, there's no hope of it getting in due to
> platform dependant behaviour.

I think it would be best if we defined an internal API for plugging in 
various kinds of locale support.  Then you can hook in this 
"newlocale", the Windows variant, ICU, or plain-old POSIX locale 
support for backward compatibility.  You already identified most of the 
API functions.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
       choose an index scan if your joining column's datatypes do not
       match

From pgsql-hackers-owner+M72592@postgresql.org Sun Sep  4 18:15:47 2005
Return-path: <pgsql-hackers-owner+M72592@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84MFk103432
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 18:15:46 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id E623DB21FBF;
	Sun,  4 Sep 2005 22:15:40 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id D671BD83B4
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 19:06:26 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 56030-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 22:06:23 +0000 (GMT)
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
	by svr1.postgresql.org (Postfix) with ESMTP id 44141D6F63
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 19:06:21 -0300 (ADT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84M6Boq008947;
	Sun, 4 Sep 2005 18:06:12 -0400 (EDT)
To: Greg Stark <gsstark@mit.edu>
cc: Martijn van Oosterhout <kleptog@svana.org>, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch 
In-Reply-To: <87u0h1ygu5.fsf@stark.xeocode.com> 
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com>
Comments: In-reply-to Greg Stark <gsstark@mit.edu>
	message dated "03 Sep 2005 17:36:02 -0400"
Date: Sun, 04 Sep 2005 18:06:11 -0400
Message-ID: <8946.1125871571@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Greg Stark <gsstark@mit.edu> writes:
> But there's nothing wrong with saying "it's slow because your
> libc is slow. Compile with this freely available library which has a better
> implementation".

The hole in that argument is the assumption that there *is* a freely
available library that can be used (where freely == BSD license).
We wouldn't be having this discussion if we knew of one.

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72596@postgresql.org Sun Sep  4 19:27:26 2005
Return-path: <pgsql-hackers-owner+M72596@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84NRQ109968
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 19:27:26 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id A427211ED4E0;
	Sun,  4 Sep 2005 23:27:20 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 64BFED8B2F
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 20:15:22 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 08469-05
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 23:15:21 +0000 (GMT)
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
	by svr1.postgresql.org (Postfix) with ESMTP id CE6D9D8AD4
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 20:15:19 -0300 (ADT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84NFLVH009399;
	Sun, 4 Sep 2005 19:15:21 -0400 (EDT)
To: Peter Eisentraut <peter_e@gmx.net>
cc: pgsql-hackers@postgresql.org, Martijn van Oosterhout <kleptog@svana.org>
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch 
In-Reply-To: <200509041906.57721.peter_e@gmx.net> 
References: <20050902130420.GA15466@svana.org> <200509041906.57721.peter_e@gmx.net>
Comments: In-reply-to Peter Eisentraut <peter_e@gmx.net>
	message dated "Sun, 04 Sep 2005 19:06:57 +0200"
Date: Sun, 04 Sep 2005 19:15:21 -0400
Message-ID: <9398.1125875721@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Peter Eisentraut <peter_e@gmx.net> writes:
> I think it would be best if we defined an internal API for plugging in 
> various kinds of locale support.

Agreed ...

> Then you can hook in this 
> "newlocale", the Windows variant, ICU, or plain-old POSIX locale 
> support for backward compatibility.

If plain old POSIX actually did what we needed, we likely wouldn't be
having this discussion at all.  POSIX doesn't give us enough visibility
of the locale's properties (in particular, which character set encoding
it wants).  The performance penalties it imposes are pretty bad also,
though arguably secondary.

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72599@postgresql.org Sun Sep  4 20:01:47 2005
Return-path: <pgsql-hackers-owner+M72599@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j8501k113218
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 20:01:46 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 5FCFD11ED4E1;
	Mon,  5 Sep 2005 00:01:40 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 3CE03D8A88
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 20:53:09 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 14738-01
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 23:53:08 +0000 (GMT)
Received: from gateway.pjmodos.bounceme.net (gprsplusa185.isp.t-mobile.cz [62.141.20.185])
	by svr1.postgresql.org (Postfix) with ESMTP id B5606D8A4E
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 20:53:03 -0300 (ADT)
Received: from modos ([10.12.0.96] ident=PJMODOS)
	by gateway.pjmodos.bounceme.net with esmtp (Exim 3.35 #1 (Debian))
	id 1EC4IE-0004Le-00; Mon, 05 Sep 2005 01:52:58 +0200
Message-ID: <431B88CD.8050904@seznam.cz>
Date: Mon, 05 Sep 2005 01:52:45 +0200
From: Petr Jelinek <pjmodos@seznam.cz>
User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206)
X-Accept-Language: cs, en, en-us
MIME-Version: 1.0
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com> <8946.1125871571@sss.pgh.pa.us>
In-Reply-To: <8946.1125871571@sss.pgh.pa.us>
Content-Type: text/plain; charset=windows-1250; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=1.173 required=5 tests=[AWL=-0.253,
	DNS_FROM_RFC_POST=1.376, FORGED_RCVD_HELO=0.05]
X-Spam-Level: *
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Tom Lane wrote:
> 
> The hole in that argument is the assumption that there *is* a freely
> available library that can be used (where freely == BSD license).
> We wouldn't be having this discussion if we knew of one.

I see this discussion as another reason to use ICU, I mean complete 
rewrite of locale handling to use ICU on all platforms. I know it's big 
project but it's doable for 8.2 and it would virtually solve all locale 
problems and could be base for new unicode/locale features. I am not 
sure if this is the way postgres wants to go tho (having dependency on 
such a big and uncommon library).

-- 
Regards
Petr Jelinek (PJMODOS)


---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

From pgsql-hackers-owner+M72602@postgresql.org Mon Sep  5 04:02:37 2005
Return-path: <pgsql-hackers-owner+M72602@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j8582Z126563
	for <pgman@candle.pha.pa.us>; Mon, 5 Sep 2005 04:02:36 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 6640811EB014;
	Mon,  5 Sep 2005 08:02:28 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 1FC36D7841
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Mon,  5 Sep 2005 04:52:11 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 09506-02
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Mon,  5 Sep 2005 07:52:06 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 0DC42D77AE
	for <pgsql-hackers@postgresql.org>; Mon,  5 Sep 2005 04:52:06 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ECBlg-0001TS-00; Mon, 05 Sep 2005 17:51:52 +1000
Date: Mon, 5 Sep 2005 09:51:52 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Greg Stark <gsstark@mit.edu>, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
Message-ID: <20050905075152.GA5278@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com> <8946.1125871571@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="MGYHOYXEY6WxJCY8"
Content-Disposition: inline
In-Reply-To: <8946.1125871571@sss.pgh.pa.us>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--MGYHOYXEY6WxJCY8
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sun, Sep 04, 2005 at 06:06:11PM -0400, Tom Lane wrote:
> Greg Stark <gsstark@mit.edu> writes:
> > But there's nothing wrong with saying "it's slow because your
> > libc is slow. Compile with this freely available library which has a be=
tter
> > implementation".
>=20
> The hole in that argument is the assumption that there *is* a freely
> available library that can be used (where freely =3D=3D BSD license).
> We wouldn't be having this discussion if we knew of one.

1. Use the one in Darwin just for the *BSDs and Solaris at least. It's
not great but it would probably work.

2. Long term, transition to ICU (http://icu.sourceforge.net/) which is
the cross-platform internationalisation library used by Java. Looks
like Mono and Gnome/GTK are going to use this (or at least allow use
of) soon also. It uses the X licence AFAICS. It's a big pill right now
but it a year it could be installed standard on most linux systems.
It's at least avaiable everywhere now.

Note, it's not compatable with POSIX locales so if we go there it'll be
an all or nothing switch. But if we intend to go there eventually, it
makes fiddling on our own library a waste of time.

Incidently, I played with the code in Darwin and getting it to compile
on a system that already has extended locale support is, uh, tricky to
say the least. Lots of conflicting definitions.
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--MGYHOYXEY6WxJCY8
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDG/kXIB7bNG8LQkwRAijSAJ4xv5JiigSwEe4jgBmxvqHN+GuOgQCfSEHp
C5tpavd58A8VEnf/BBUgfzo=
=gsNf
-----END PGP SIGNATURE-----

--MGYHOYXEY6WxJCY8--

From pgsql-hackers-owner+M72586@postgresql.org Sun Sep  4 09:37:44 2005
Return-path: <pgsql-hackers-owner+M72586@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84Dbh100551
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 09:37:43 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 824CCB21F00;
	Sun,  4 Sep 2005 13:37:37 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 413CCD7015
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 10:26:10 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 20529-10
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 13:26:07 +0000 (GMT)
Received: from sraigw.sra.co.jp (sraigw.sra.co.jp [202.32.10.2])
	by svr1.postgresql.org (Postfix) with ESMTP id 12EF2D6F75
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 10:26:06 -0300 (ADT)
Received: from srascb.sra.co.jp (srascb [133.137.8.65])
	by sraigw.sra.co.jp (Postfix) with ESMTP
	id 08EB262112; Sun,  4 Sep 2005 22:26:09 +0900 (JST)
Received: from srascb.sra.co.jp (localhost [127.0.0.1])
	by localhost.sra.co.jp (Postfix) with ESMTP
	id D1D9610CD06; Sun,  4 Sep 2005 22:26:09 +0900 (JST)
Received: from sranhm.sra.co.jp (sranhm [133.137.44.16])
	by srascb.sra.co.jp (Postfix) with ESMTP
	id ABF1210CD04; Sun,  4 Sep 2005 22:26:09 +0900 (JST)
Received: from localhost (sraihb-hub.sra.co.jp [133.137.8.6])
	by sranhm.sra.co.jp (8.9.3+3.2W/3.7W-srambox) with ESMTP id WAA12274;
	Sun, 4 Sep 2005 22:26:09 +0900
Date: Sun, 04 Sep 2005 22:25:36 +0900 (JST)
Message-ID: <20050904.222536.39155679.ishii@sraoss.co.jp>
To: kleptog@svana.org
cc: tgl@sss.pgh.pa.us, gsstark@mit.edu, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions
From: Tatsuo Ishii <t-ishii@sra.co.jp>
In-Reply-To: <20050903203434.GA4281@svana.org>
References: <87k6hzzems.fsf@stark.xeocode.com>
	<14696.1125675741@sss.pgh.pa.us>
	<20050903203434.GA4281@svana.org>
X-Mailer: Mew version 2.3 on Emacs 20.7 / Mule 4.1
	=?iso-2022-jp?B?KBskQjAqGyhCKQ==?=
MIME-Version: 1.0
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.028 required=5 tests=[AWL=0.028]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

> 3. Compiled locale files are large. One UTF-8 locale datafile can
> exceed a megabyte. Do we want the option of disabling it for small
> systems?

To avoid the problem, you could dynmically load the compiled
tables. The charset conversion tables are handled similar way.

Also I think it's important to allow user defined collate data. To
implement the CREATE COLLATE syntax, we need to have that capability
anyway.

> 4. Do we want the option of running system locale in parallel with the
> internal ones?
> 
> 5. I think we're going to have to deal with the very real possibility
> that our locale database will not be as good as some of the system
> provided ones. The question is how. This is quite unlike timezones
> which are quite standardized and rarely change. That database is quite
> well maintained.
> 
> Would people object to a configure option that selected:
>   --with-locales=internal     (use pg database)
>   --with-locales=system       (use system database for win32, glibc or MacOS X)
>   --with-locales=none         (what we support now, which is neither)
> 
> I don't think it will be much of an issue to support this, all the
> functions take the same parameters and have almost the same names.

To be honest, I don't understand why we have to rely on (often broken)
system locales. I don't think building our own locale data is too
hard, and once we make up it, the maintenace cost will be very small
since it should not be changed regularly. Moreover we could enjoy the
benefit that PostgreSQL handles collations in a corret manner on any
platform which PostgreSQL supports.

> 6. Locales for SQL_ASCII. Seems to me you have two options, either
> reject COLLATE altogether unless they specify a charset, or don't care
> and let the user shoot themselves in the foot if they wish...
> 
> BTW, this MacOS locale supports seems to be new for 10.4.2 according to
> the CVS log info, can anyone confirm this?
> 
> Anyway, I hope this post didn't bore too much. Locale support has been
> one of those things that has bugged me for a long time and it would be
> nice if there could be some real movement.

Right. We Japanese (and probably Chinese too) have been bugged by the
broken mutibyte locales for long time. Using C locale help us to a
certain extent, but for Unicode we need correct locale data, othewise
the sorted data will be completely chaos.
--
SRA OSS, Inc. Japan
Tatsuo Ishii

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72587@postgresql.org Sun Sep  4 11:13:07 2005
Return-path: <pgsql-hackers-owner+M72587@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84FD5107053
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 11:13:06 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id D1EC5B21F00;
	Sun,  4 Sep 2005 15:13:04 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 8823DD723A
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 12:01:47 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 65135-01
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 15:01:43 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 5896DD70F7
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 12:01:40 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1EBvzd-0006eP-00; Mon, 05 Sep 2005 01:01:13 +1000
Date: Sun, 4 Sep 2005 17:01:13 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Tatsuo Ishii <t-ishii@sra.co.jp>
cc: tgl@sss.pgh.pa.us, gsstark@mit.edu, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions
Message-ID: <20050904150055.GB21198@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <20050904.222536.39155679.ishii@sraoss.co.jp>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="ZfOjI3PrQbgiZnxM"
Content-Disposition: inline
In-Reply-To: <20050904.222536.39155679.ishii@sraoss.co.jp>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--ZfOjI3PrQbgiZnxM
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sun, Sep 04, 2005 at 10:25:36PM +0900, Tatsuo Ishii wrote:
> > 3. Compiled locale files are large. One UTF-8 locale datafile can
> > exceed a megabyte. Do we want the option of disabling it for small
> > systems?
>=20
> To avoid the problem, you could dynmically load the compiled
> tables. The charset conversion tables are handled similar way.

That's not the point, ofcourse they are loaded dynamically. The
question is, when do we create the files in the first place. There are
48*15 =3D 750 combinations which would amount to tens of megabytes of
essentially useless data. *When* you create the files is an important
question. Compile time is out.

Charset conversion is completely different, there just arn't that many
combinations.

> Also I think it's important to allow user defined collate data. To
> implement the CREATE COLLATE syntax, we need to have that capability
> anyway.

Most OS's allow you to create collate data yourself anyway, why do we
need to implement this too?

> To be honest, I don't understand why we have to rely on (often broken)
> system locales. I don't think building our own locale data is too
> hard, and once we make up it, the maintenace cost will be very small
> since it should not be changed regularly. Moreover we could enjoy the
> benefit that PostgreSQL handles collations in a corret manner on any
> platform which PostgreSQL supports.

You say building our own locale data is not hard. I disagree, it's a
waste of time we can do without. Unless you know the language yourself
you cannot check changes made by anybody else. If there's an error in
locale ordering, take it up with your OS distributor.

I also think we open ourselves to questions like:

1. My locale is supported by the system but not by PostgreSQL, why?
2. My locale was supported last release but not this one, why?
3. Why does PostgreSQL sort differently from 'sort' or any other app on
my system?

> Right. We Japanese (and probably Chinese too) have been bugged by the
> broken mutibyte locales for long time. Using C locale help us to a
> certain extent, but for Unicode we need correct locale data, othewise
> the sorted data will be completely chaos.

Ok, is glibc still wrong or are they just implementing the unicode
standard and that's what's wrong.

All I'm saying is that we need to allow use of system locales until our
native locale support is mature. In the end something like ICU
(http://icu.sourceforge.net/) will end up obsoleting us. Nobody (in
free-software anyway) uses it yet, but eventually it may be viable to
require that to allow system independant locales.
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--ZfOjI3PrQbgiZnxM
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDGwwmIB7bNG8LQkwRAmKxAJ9MccMVTLVLqW7c9Lsa8FzqEMvNGgCdGFgg
nsvl/ZIAK/Qe1XpkMOItRFM=
=h0V5
-----END PGP SIGNATURE-----

--ZfOjI3PrQbgiZnxM--

From pgsql-hackers-owner+M72590@postgresql.org Sun Sep  4 15:15:31 2005
Return-path: <pgsql-hackers-owner+M72590@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84JFU107171
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 15:15:31 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 950F5B21FBE;
	Sun,  4 Sep 2005 19:15:26 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id AA4C5D85BF
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 16:02:50 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 51227-06
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 19:02:48 +0000 (GMT)
Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227])
	by svr1.postgresql.org (Postfix) with ESMTP id CC650D8580
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 16:02:46 -0300 (ADT)
Received: from localhost ([127.0.0.1] helo=stark.xeocode.com)
	by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian))
	id 1EBzlN-0003LY-00; Sun, 04 Sep 2005 15:02:45 -0400
To: Tatsuo Ishii <t-ishii@sra.co.jp>
cc: kleptog@svana.org, tgl@sss.pgh.pa.us, gsstark@mit.edu,
   pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions
References: <87k6hzzems.fsf@stark.xeocode.com>
	<14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org>
	<20050904.222536.39155679.ishii@sraoss.co.jp>
In-Reply-To: <20050904.222536.39155679.ishii@sraoss.co.jp>
From: Greg Stark <gsstark@mit.edu>
Organization: The Emacs Conspiracy; member since 1992
Date: 04 Sep 2005 15:02:45 -0400
Message-ID: <87fysky7u2.fsf@stark.xeocode.com>
Lines: 22
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


Tatsuo Ishii <t-ishii@sra.co.jp> writes:

> To be honest, I don't understand why we have to rely on (often broken)
> system locales. I don't think building our own locale data is too
> hard, and once we make up it, the maintenace cost will be very small
> since it should not be changed regularly. Moreover we could enjoy the
> benefit that PostgreSQL handles collations in a corret manner on any
> platform which PostgreSQL supports.

I think it's sheer madness to try to reproduce large swaths of the OS inside
Postgres because you're unhappy with the quality of the OS implementation. You
should be asking yourself why OS vendors have such a hard time getting this
stuff right and why would Postgres do any better. Wouldn't that work be better
spent improving the database functionality of Postgres?

Or at least better spent improving the locale support for the entire OS? It
would be positively awful if every application on my system had its own locale
database each of which had its own set of bugs and its own feature set.

-- 
greg


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq

From pgsql-hackers-owner+M72597@postgresql.org Sun Sep  4 19:28:34 2005
Return-path: <pgsql-hackers-owner+M72597@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84NSY110074
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 19:28:34 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 7898611ED4E0;
	Sun,  4 Sep 2005 23:28:28 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id BE640D8910
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 20:19:37 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 09430-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 23:19:37 +0000 (GMT)
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
	by svr1.postgresql.org (Postfix) with ESMTP id 7DE22D8864
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 20:19:34 -0300 (ADT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84NJclc009441;
	Sun, 4 Sep 2005 19:19:38 -0400 (EDT)
To: Greg Stark <gsstark@mit.edu>
cc: Tatsuo Ishii <t-ishii@sra.co.jp>, kleptog@svana.org,
   pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions 
In-Reply-To: <87fysky7u2.fsf@stark.xeocode.com> 
References: <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <20050904.222536.39155679.ishii@sraoss.co.jp> <87fysky7u2.fsf@stark.xeocode.com>
Comments: In-reply-to Greg Stark <gsstark@mit.edu>
	message dated "04 Sep 2005 15:02:45 -0400"
Date: Sun, 04 Sep 2005 19:19:38 -0400
Message-ID: <9440.1125875978@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Greg Stark <gsstark@mit.edu> writes:
> I think it's sheer madness to try to reproduce large swaths of the OS
> inside Postgres because you're unhappy with the quality of the OS
> implementation. You should be asking yourself why OS vendors have such
> a hard time getting this stuff right

In the case of the *BSDs, it's pretty obviously because they don't care.

> and why would Postgres do any better

In the first place, we do care, and in the second place, having to deal
with only one set of locale bugs would in itself be a huge advance over
where we are now.

We went over to maintaining our own timezone code for more or less the
same reasons, and in hindsight that was obviously the right decision.
Locale support is a bigger chunk, no doubt about it, but we also have
a lot of motivation.

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

From pgsql-hackers-owner+M72619@postgresql.org Mon Sep  5 18:14:28 2005
Return-path: <pgsql-hackers-owner+M72619@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j85MER115087
	for <pgman@candle.pha.pa.us>; Mon, 5 Sep 2005 18:14:27 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 68D9F11EB015;
	Mon,  5 Sep 2005 22:14:19 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 92268D6FB8
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Mon,  5 Sep 2005 18:59:42 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 29325-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Mon,  5 Sep 2005 21:59:38 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 98C2CD7A97
	for <pgsql-hackers@postgresql.org>; Mon,  5 Sep 2005 18:59:33 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ECP02-0004SQ-00; Tue, 06 Sep 2005 07:59:34 +1000
Date: Mon, 5 Sep 2005 23:59:33 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] Install Darwin's locale library on your system :)
Message-ID: <20050905215928.GB5278@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="hHWLQfXTYDoKhP50"
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--hHWLQfXTYDoKhP50
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Well, it was pointed out the other day that the Darwin C library
supports the non-standard extensions to the POSIX locale interface and
that this might be ported to other systems so PostgreSQL could use it.

So, I have written a few scripts which download the libc and locale
library from darwinsource, shuffle some files around and build the
result into a library called libdummylocale.so. It basically completely
replaces your locale support on whatever system you use it on.

It's all under the APSL, though some parts may be BSD licenced.

Let me say right now, the locale support here sucks, no two ways about
it. It doesn't support a single UTF-8 locale. Oh, it lets you specify
them, but when you ask for the CHARSET it still says US-ASCII. It does
support a number of other different charsets. (Not for collation
though).

So my challenge to those people who think maintaining a locale library
is easy: make *one* locale in FreeBSD (or Darwin or this lib) support
full UTF-8 collation in whatever locale and/or charset you choose. It's
all downhill from there.

While it builds simple programs, I don't think it's totally safe. You'd
need to rename the headers at least. And building on Darwin will
probably blow up due to the way it plays fast and loose with Darwin
specific #defines. But it's a beginning if anyone is interested. It
builds in my glibc system.

I'm going to drop the idea of making a locale library, there's just
nothing good enough. glibc is the only thing that comes close. From here
on I'm going to work on COLLATE for systems that support xlocale, with
an eye on ICU if/when it becomes standard enough.

Download: http://svana.org/kleptog/pgsql/dummylocale.tar.gz

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--hHWLQfXTYDoKhP50
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDHL++IB7bNG8LQkwRAjvyAJ98Hwm6Ak7zSdEU7shrzxbvMJJp/QCfcJSq
ZseeCwJGWAs+3MEPEr8u2dM=
=rr4R
-----END PGP SIGNATURE-----

--hHWLQfXTYDoKhP50--

From pgsql-patches-owner+M17366@postgresql.org Wed Sep  7 12:16:02 2005
Return-path: <pgsql-patches-owner+M17366@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j87GG1104073
	for <pgman@candle.pha.pa.us>; Wed, 7 Sep 2005 12:16:01 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id D3FD711EB015;
	Wed,  7 Sep 2005 16:15:58 +0000 (GMT)
X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 326F8D7B55
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>; Wed,  7 Sep 2005 13:11:41 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 66946-04
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>;
	Wed,  7 Sep 2005 16:11:25 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id D88A6D6E3B
	for <pgsql-patches@postgresql.org>; Wed,  7 Sep 2005 13:11:20 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ED2W5-00052g-00; Thu, 08 Sep 2005 02:11:17 +1000
Date: Wed, 7 Sep 2005 18:11:17 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: pgsql-patches@postgresql.org
Subject: [PATCHES] For review: Initial support for COLLATE
Message-ID: <20050907161112.GA10273@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="ibTvN161/egqYuK8"
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-patches
List-Archive: <http://archives.postgresql.org/pgsql-patches>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-patches.postgresql.org>
List-Owner: <mailto:pgsql-patches-owner@postgresql.org>
List-Post: <mailto:pgsql-patches@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-patches>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-patches>
Precedence: bulk
Sender: pgsql-patches-owner@postgresql.org
Status: OR


--ibTvN161/egqYuK8
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

[Please CC any replies, thanks]

This patch is the beginnings of support for COLLATE. I need to do some
other work for a few days so I'm posting here to get some initial
reviews. Various parts are marked [done] and [not done].

The steps involved are:

- Add COLLATE to grammer as part of expression tree. New CollateClause
  node stores the relevent facts. [done]

- parse_expr goes through tree and determines the appropriate COLLATE
  state for each node as per SQL spec. [done]

  Note: PostgreSQL doesn't really have a way to identify text-like
  types and there's no real need to anyway. The implementation allows
  any node to have a collate state. This can be used for other things,
  see below.

- CREATE COLLATE statement [not done, currently added on the fly]

- Two new datatypes, pg_locale_t and pg_localedata_t. The former
  represents a locale (and will eventually have an OID). The latter is
  an anonymous cookie that stores locale specific information and is
  passed to functions that need it. It may be that once done, this
  latter type will vanish again. [done]

- Several utility functions in the new file pg_xlocale.c for use by the
  rest of the system. [done]

- Add boolean column 'proislocalized' to pg_proc which indicates if the
  output of this function is affected by the LOCALE. eg textcat doesn't
  care but textle does. [done] This is for:

  a) So the parser can complain about functions that look at the
     locale/collate order but it's not clear from the arguments (state
     None per SQL spec) and it's not specified explicitly. [not done]

  b) So when a column or function is indexed, the index code knows if
     the locale is relevent to sorting order. This is particularly
     interesting for btree indexes on character strings. [not done]

  Currently I've marked 58 functions as being locale sensetive, but
  that list will need careful going over. To some extent it can be
  checked automatically by examining which backend functions use the
  new PG_GETLOCALE() macro.

- Check for correct encoding in loaded locales [not done]

- Check the partial indexes do the right thing when matching
  expressions. [not done]

- Docs, regression, etc...

- make check: right now I'm getting some regressions in the rules
  and plpgsql, very odd... Possibly due to the fact that rules get
  collate nodes with locales that don't persist across invokations.

Goals:

- This setup extends the SQL spec a bit, in the sense that COLLATE can
  be attached to anything. It is my intention to allow functions such
  as to_char() and to_timestamp() to be localized. eg:

test=3D# select cash_out('1.00'::money collate 'nl_NL'), cash_out('1.00'::m=
oney collate 'en_AU');
 cash_out | cash_out=20
----------+----------
 EUR1,00  | $1.00
(1 row)

- Should LOCALE be created as a synonym for COLLATE? It reads more
  naturally.

- Currently LC_COLLATE is fixed at initdb and LC_NUMERIC and
  LC_CURRENCY can be altered. The idea is that eventually even
  LC_COLLATE can be altered anytime (from the users point view anyway),
  as any objects that care will store the collate order they want and
  can execute functions as appropriate.

- Eventually once the transition to full locale support is complete,
  change the backend so it always runs under locale C and only
  functions on userdata are affected by locales. This means that
  unquoted identifiers when converted to lowercase will be lowered as
  per ASCII rules. Judging by [1] this is what people want, but
  feedback would be nice.

[1] http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg39742.html

  Hence, right now, names compare using normal strcmp, but text,
  varchar, etc use strcoll.

- The patch as it currently stands won't compile at all without system
  xlocale support. The goal is to provide some level of backward
  compatability where the COLLATE clause can be used, but only to
  affect functions like to_char(). Changing COLLATE order would be
  forbidden (ie just like now).

- Eventually, once the parts relevent to locales are sufficiently
  abstracted, look into ICU to plug it in. Unfortunatly, it's a
  completely different model (all utf-16) so that's another phase
  altogether.

- The default type output functions should never be locale specific.
  This is to avoid issues with pgdump and frontends. Create a function
  'localize(anyelement)' to "do the obvious" to force it to happen.

Download:=20
  Compressed 57K, uncompressed >500K but that's due to rewriting the
  whole of pg_proc. The important code is not so big. Against todays
  CVS.

http://svana.org/kleptog/pgsql/collate2.patch.gz

Examples:

test=3D# SELECT text('a') < text('B') COLLATE 'C', text('a') < text('B') CO=
LLATE 'en_US.UTF-8';
 ?column? | ?column?=20
----------+----------
 f        | t
(1 row)

test=3D# SELECT text('A') < text('b') COLLATE 'C', text('A') < text('b') CO=
LLATE 'en_US.UTF-8';
 ?column? | ?column?=20
----------+----------
 t        | t
(1 row)

test=3D# SELECT text('A') COLLATE 'en_US.UTF-8' < text('b') COLLATE 'C';
ERROR:  Conflicting COLLATE clauses

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--ibTvN161/egqYuK8
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDHxEgIB7bNG8LQkwRAqTqAJ0TOH3RkxnYHJmCPhjjYha0idsEoACfTjY0
EP4a7CtIaZ0Ar0dyuPja3h8=
=8zan
-----END PGP SIGNATURE-----

--ibTvN161/egqYuK8--

From pgsql-patches-owner+M17368@postgresql.org Wed Sep  7 16:31:39 2005
Return-path: <pgsql-patches-owner+M17368@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j87KVd101436
	for <pgman@candle.pha.pa.us>; Wed, 7 Sep 2005 16:31:39 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 847AF11EB01E;
	Wed,  7 Sep 2005 20:31:32 +0000 (GMT)
X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 22823D7D5C
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>; Wed,  7 Sep 2005 16:12:20 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 19251-09
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>;
	Wed,  7 Sep 2005 19:12:13 +0000 (GMT)
Received: from mail.gmx.net (mail.gmx.de [213.165.64.20])
	by svr1.postgresql.org (Postfix) with SMTP id 05BA7D7B7E
	for <pgsql-patches@postgresql.org>; Wed,  7 Sep 2005 16:12:13 -0300 (ADT)
Received: (qmail invoked by alias); 07 Sep 2005 19:12:14 -0000
Received: from dsl-082-083-195-234.arcor-ip.net (EHLO colt.pezone.net) [82.83.195.234]
  by mail.gmx.net (mp022) with SMTP; 07 Sep 2005 21:12:14 +0200
X-Authenticated: #495269
From: Peter Eisentraut <peter_e@gmx.net>
To: Martijn van Oosterhout <kleptog@svana.org>
Subject: Re: [PATCHES] For review: Initial support for COLLATE
Date: Wed, 7 Sep 2005 21:12:12 +0200
User-Agent: KMail/1.8.1
References: <20050907161112.GA10273@svana.org>
In-Reply-To: <20050907161112.GA10273@svana.org>
cc: pgsql-patches@postgresql.org
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-ID: <200509072112.12955.peter_e@gmx.net>
X-Y-GMX-Trusted: 0
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.044 required=5 tests=[AWL=-0.006,
	FORGED_RCVD_HELO=0.05]
X-Mailing-List: pgsql-patches
List-Archive: <http://archives.postgresql.org/pgsql-patches>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-patches.postgresql.org>
List-Owner: <mailto:pgsql-patches-owner@postgresql.org>
List-Post: <mailto:pgsql-patches@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-patches>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-patches>
Precedence: bulk
Sender: pgsql-patches-owner@postgresql.org
Status: OR

Martijn van Oosterhout wrote:
> - Should LOCALE be created as a synonym for COLLATE? It reads more
>   naturally.

No, and in fact the terminology mixup in your patch and description 
concerns me.  If you are talking about collation, then the data types, 
system catalog columns, etc. should talk about collation, not about 
"locale", because that encompasses a number of other things that can be 
handled independent of the collation order.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
       choose an index scan if your joining column's datatypes do not
       match

From pgsql-hackers-owner+M77964=pgman=candle.pha.pa.us@postgresql.org Tue Dec 27 06:30:14 2005
Return-path: <pgsql-hackers-owner+M77964=pgman=candle.pha.pa.us@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jBRDUET16504
	for <pgman@candle.pha.pa.us>; Tue, 27 Dec 2005 08:30:14 -0500 (EST)
Received: from postgresql.org (postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 3BCBA67A57F
	for <pgman@candle.pha.pa.us>; Tue, 27 Dec 2005 09:30:14 -0400 (AST)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by postgresql.org (Postfix) with ESMTP id 1DB3B9DC859
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Tue, 27 Dec 2005 09:29:47 -0400 (AST)
Received: from postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 89101-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Tue, 27 Dec 2005 09:29:50 -0400 (AST)
X-Greylist: from auto-whitelisted by SQLgrey-
Received: from svana.org (svana.org [203.20.62.76])
	by postgresql.org (Postfix) with ESMTP id 319839DC8A5
	for <pgsql-hackers@postgresql.org>; Tue, 27 Dec 2005 09:29:44 -0400 (AST)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ErEte-0001lm-00; Wed, 28 Dec 2005 00:29:46 +1100
Date: Tue, 27 Dec 2005 14:29:46 +0100
From: Martijn van Oosterhout <kleptog@svana.org>
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] Proposed COLLATE implementation
Message-ID: <20051227132941.GA32404@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="W/nzBZO5zC0uMSeA"
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, score=0.08 required=5 tests=[AWL=0.080]
X-Spam-Score: 0.08
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--W/nzBZO5zC0uMSeA
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings all,

If you're not interested in COLLATE, operator classes or related
things, stop now, this is quite a long email.

Firstly, status. PostgreSQL doesn't really support collations at all.
The order of strings is defined at initdb time by the locale then and
cannot be changed later. We allow lists to be sorted in either
ascending or decending order but that's about it. Whatever order there
is is deduced from b-tree operator classes.

The purpose of this patch is to raise collations to (reasonably) first
class object. The idea is that you can define a collation across any
type and that you will then be able to ORDER BY, GROUP BY and INDEX
using that collation. A collation defines both order and equality.

The SQL standard does define COLLATE although they only apply that to
character strings. There are no predefined collations in the standard.
There are rules about how collations and collation states propegate
from the leaves of the parse tree all the way to the root. In its
simplest form, columns and constants have defined collations which
modify the behaviour of functions using these values. At any point in
the parse tree the user can override the collation with an explicit
<collate clause>. If there is ambiguity about what collation applies at
any point for a function that needs to know, this is a error.

All this is parse-time analysis.

Proposed Implementation:

NODES

To implement the above, two new node types are created: CollateClause
which represents the <collate clause> in SQL syntax, and CollateState
which represents the actual state at any node. Currently the only nodes
expected to require these are OpExpr, FuncExpr, Var and Const. Although
I guess it may apply to any node that can be used in an expression.

CATALOG CHANGES

To track collations requires a new table in the catalog, which I have
named pg_collations. It contains the following fields:

Oid oid;           -- OID for this collation
Name collname;     -- Name of the collation (for collate clause)
bool collasc;      -- Ascending or descending
Oid  collopclass;  -- Implementing Operator Class
int4 colltype;     -- Currently, 0=3Dsimple, 1=3Duses locale
Oid colllocale;    -- Locale in pg_locales

(Should we be identifying the type here? or is it ok to lookup the type
via the operator class).

The first few fields name the collation so it can be referred to by the
user. Then the collasc field determines how to use the operator class
as given in the collopclass field. If it indicates descending order, it
will invert the sense of the operator class. For example, asking for
the GT op for a reverse collation will actually return the LT operator
for the operator class.

The purpose of the colltype and colllocale fields are described further
down.

The important thing at this point is that by specifying a collation you
are also specifying an operator class. At the moment the ascending and
descending collations for each type are hard-coded for initdb. At the
moment they have been allocated OIDs starting at 2800, which is the
first large available block.

Each column of a table has a default collation, which defaults to the
default collation of the type but can be specified in the table
declaration. To store this requires an additional column in
pg_attribute (attcollate) which contains the OID of the collation for
that column. When it is referenced in a query, this collation is copied
to the CollateState node of the Var node, from whence it can affect the
query.

Finally, to allow the parser to complain about ambiguous CollateStates,
we need to indicate which functions actually need a senseble collate
state to function. To this end a single boolean field has been added to
pg_proc (proneedcollate). If this is true, the parser should error out
when the collation state is COLLATE_NONE.

INDEXES

Another place you will be allowed to use the collate clause is while
creating indexes. If you declare an index using a particular collation,
it can be used in queries that order by the same collation. Note, that
the collate clause indicates the operator class, so you can either
specify one or the other, but not both.

So each column of an index will also have a collation. However,
pg_attribute has already got an extra field to store the collation for
columns so it makes sense to store the collation here. In the process
the pg_index.indclass field becomes redundant as it can be inferred
from the pg_attribute rows associated with the index.

To make this work there also needs to be a notion of compatability
between collations. For example, two collations which are the reverse
of eachother are compatable in the sense that an index defined with one
collation would be usable for the other simply by scanning in reverse.

FUNCTIONS

In particular for string comparison but also possibly for user-defined
types, a function will need to know what collation it is operating
under. For this purpose an extra field (fn_collate) is added to
FmgrInfo which is filled in with the collation from the expression tree
(if any) or wherever relevent (eg. from the pg_attribute column when
doing statistics or creating indexes).

A PG_GETCOLLATE() macro is added to facilitate user-functions
retreiving this data. This function throws an error when no collation
has been defined. This shouldn't happen in practice as issues should
have been weeded out at parse-time.

This macro returns the OID of the collation but in many cases it will
not be necessary. In particular, functions should NOT invert their
result if the collation is inverted. It is considered the
responsibility of the caller to invert the result if necessary. The
reasons for this are:

1. In most cases that matter (order comparison) the issue can be dealt
with at parse time by the NEGATOR or COMMUTATOR options.
2. For index scans, we would just do a reverse scan instead (or forward
if the index is inverted)
3. Requiring every function to check the collation for inversion is
wasteful, since in many cases the case can be dealt with statically.

DEFAULT COLLATIONS

At this point I'm inclined to define a few collations to be built in or
specially handled:

COLLATE ASC    - default collation for type, ascending (ie, what we do now)
COLLATE DESC   - default collation for type, inverted
COLLATE POSIX  - For strings, define a simple bytewise string comparison.

Indeed, it is expected that by default, all columns involving strings
in the system catalog will always use COLLATE POSIX. Additionaly, type
"name" will always use that collation, even if the user changes the
default (by a method to be specified). This is straightforwardly done
at initdb time.

The purpose of COLLATE DESC is to simplify index declarations. Saying

CREATE INDEX foo ON bar( a COLLATE ASC, b COLLATE DESC );

would allow it to be used in a query using ORDER BY a, b DESC, without
the user having to lookup the name of the collation.

When it comes to naming collations, the question arises whether
ascending/descending collations need to have different names. Or
should there be two collations with the same name with ASC/DESC as a
modifier? Do collations have to be unique across different types; for
example, can varchar and text both have a collation "ignorecase"?

Another issue is that a column could be declared with a descending
collation by default. Say it was an integer column, then (a < 5) would
return FALSE for a =3D 1. While technically correct, I'm thinking of
ruling it out on the basis of being too confusing, and only allow
descending collations at query time or in index specifications.

Another strange point at the moment is how to determine the default
collation of a type. At the moment it is done by finding the default
operator class and looking up the ascending version of that. However,
we may want to provide the user a way of specifying it directly,
perhaps by:

ALTER TYPE text SET DEFAULT COLLATION ignorecase;

PATHKEYS

Currently during planning, pathkeys are indicated by an operator of the
operator class. Here we would simply replace that with the oid of the
collation, which can be matched directly with the collation defined by
the index.

USER DEFINED TYPES

None of this is interesting unless it can be applied to user-defined
types also. Fortunatly this is easy, when the user declares a b-tree
operator class, we can generate the collations automatically. We may
even allow the user to specify the name of the collation. However, if
the user wanted to create multiple collations based on the same
operator class (by using the PG_GETCOLLATE() macro above, we may want
to provide them a way of creating them directly.

COLLATIONS USING LOCALES

For strings, collation can be done in many different ways defined by
what is referred to as a locale. As indicated above in the definition
of pg_collations, there is a colltype field. For most collations this
will be 0 (simple collation). However, for strings the intention is to
use a type 1 (using locales). In this case the last column refers to
the OID of the locale, so you can many collations using the same
operator class, but different locale oids. On a system level it changes
nothing, but inside the functions implementing it, they should use
PG_GETLOCALE(). This will return an opaque pg_locale_t (see below)
handle which can then be used to implement the specifics.

In principle, user-defined types need to be able to use this also,
perhaps by using the clause COLLATE USING LOCALE in the operator class.
In theory there should a collation for each combination of
locale-dependant datatype, locale and order ascending/descending.
How/when these are created has not yet be determined.

MORE TYPES OF COLLATION

Another collation type I've speculated about but not thought about
implementing is a "mapping collation", in which you map the values
through a function and then collate that. The obvious example would be
a case-insensetive mapping where lower is applied before collation.

Implementation could be pretty much done by simply substituting the
functions into the parse tree. For example, if you defined something
like:

CREATE COLLATION ignorecase ON text USING lower($1) COLLATE defaulttext;

Then anytime you did a comparison with that collation, you would simply
insert those function calls into the parse-tree and then collate with
"defaulttext". When declaring an index you would just make it a
functional index. The rules for functional indexes should make it work
out-of-the-box.

OTHER TECHNICAL ISSUES

- Applying a COLLATE clause to an unknown literal causes it to be
coerced to the type that collation is based on. But what about if we
have something like COLLATE DESC?

- This requires some changes in the bootstrap procedures given that we
need to be able to do lookups on the operator class for each type
fairly early on. Some are predefined but it does require moving the
opclass setup further up the list. However, if we store a default
collation in pg_type, we wouldn't need to do that.

- Sorting arrays. Should they get their own collations, or should
they use the collations of their base types.

LOCALES

I've left this to the end because I don't want people distracted by
what is essentially a side-issue. While this would be needed to
implement COLLATE the way the SQL spec intended, it can actually be
implemented and dealt with as a seperate patch. The main reason a basic
implementation exists is that it provides a great way of finding places
that didn't define a collation, since any comparison involving "text"
requires one.

To deal with locales I created another table in the catalog,
pg_locales. This provides an OID which can be referenced from
elsewhere, such as the pg_collations table.

The design is intended to provide some pluggability, so locale
information can come from multiple sources. Also, each locale will be
referenced by an identifier which is unrelated to any external
identifier, so we're not bound by them.

The columns defined currently are:

Name	locname       - Identifier used by postgresql
Name	locsysname    - String identifying the locale for the locale provider
int4	locsource     - System providing this locale
int4	locencoding   - Encoding expected by provider

It is expected that the list of sources for locale data will be short,
probably hard-coded into the backend (currenty internal/system/icu).
The only locale defined at startup is POSIX, which is implemented
internally. The intention is for any other locales to be defined at the
end of initdb. The expected syntax is something like:

CREATE LOCALE hungarian AS 'hu_HU' USING glibc;

This should use the provider to check the locale exists and has a
conpatible encoding. If so it is entered into the table ready for use.

In the backend, there will be implementations of functions like
pg_strcoll_l, pg_localeconv_l, which work like the C system library
versions only they take an extra pg_locale_t argument. This is used to
dispatch the call to the right place. There will be a function to
quickly determine if a locale is C to shortcircuit complexity where it
is not needed.

STATUS

Implementation so far is available here:

http://svana.org/kleptog/temp/collate-current.patch.gz

This patch isn't "clean" and changes a few things that are not strictly
necessary. It won't finish initdb right now because it gets an error in
ANALYSE (the array issue above).

Feedback, help, comments: please reply.

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--W/nzBZO5zC0uMSeA
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDsUHDIB7bNG8LQkwRAnh0AJ0YUNLkVaSY3u0jWBPdlaq+9dujZACfTmis
JR1mF60lKx14Ih850p3lpVk=
=/Ghs
-----END PGP SIGNATURE-----

--W/nzBZO5zC0uMSeA--