Project

General

Profile

Feature #3759

setup, test and document PostgreSQL on Windows

Added by Greg Shah over 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
version_reported:
version_resolved:

default.profile.log Magnifier (8.28 KB) Ovidiu Maxiniuc, 11/23/2018 06:21 AM

mingw.profile.log Magnifier (1.25 MB) Ovidiu Maxiniuc, 11/23/2018 06:21 AM

aol.properties.mingw64 - Maven config file for MINGW64 (33.8 KB) Eugenie Lyzenko, 11/23/2018 12:24 PM

libintl.h Magnifier - Missed header file (16.3 KB) Eugenie Lyzenko, 12/10/2018 11:00 PM

aol.mingw.properties Magnifier - Custom GCC aol.properties (3.37 KB) Eugenie Lyzenko, 12/10/2018 11:00 PM

openssl.zip - OpenSSL header files needed to built PL/Java (389 KB) Ovidiu Maxiniuc, 12/12/2018 04:31 AM

History

#1 Updated by Eric Faulhaber over 5 years ago

  • Assignee set to Ovidiu Maxiniuc

#2 Updated by Ovidiu Maxiniuc over 5 years ago

A few questions here:

  • Will the database server run on same machine as the FWD server?
    Theoretically, this should not matter because we are connecting using host:port in connection string from directory.xml. The only difference would be the latency imposed by networking.
  • Is there a preferred PostgreSQL version I should focus on? I am running currently on 9.2 / 9.4 but the latest version is 11.1 (which I understand is not yet available on Ubuntu platforms).

I understand that the result will be integrated in our wiki similar to https://proj.goldencode.com/projects/p2j/wiki/Database_Server_Setup_for_PostgreSQL_on_Linux or https://proj.goldencode.com/projects/p2j/wiki/Database_Server_Setup_for_SQLServer_on_Windows.

#3 Updated by Greg Shah over 5 years ago

  • Will the database server run on same machine as the FWD server?

Yes.

  • Is there a preferred PostgreSQL version I should focus on? I am running currently on 9.2 / 9.4 but the latest version is 11.1 (which I understand is not yet available on Ubuntu platforms).

Use the same version that is the latest current version for 16.04 (9.5?) since that is the latest version that we will have tested. If there is some issue with that version on Windows, then you can drop back to an older supported version that will avoid the issue.

I'd don't want to mix up moving to a new version with the Windows change.

I understand that the result will be integrated in our wiki similar

Correct.

#4 Updated by Ovidiu Maxiniuc over 5 years ago

  • Status changed from New to WIP

I am having some difficulties with building PL/Java on Windows. Apparently, msvc profile is selected by Maven. I do not understand why. Of course it will fail because there is no 'link' in MinGW. If I manually select the 'compiler-mingw64' profile I get a minute of scrolling errors but the root cause seems to be int64 support.

#5 Updated by Greg Shah over 5 years ago

Eugenie: Please review this task and offer any advice about mingw.

#6 Updated by Ovidiu Maxiniuc over 5 years ago

Please find the attached logs from

mvn clean install > default.profile.log

and
mvn clean install -P compiler-mingw64 > mingw.profile.log

I noticed that the msvc profile has an additional include path: ${PGSQL_INCLUDEDIR-SERVER}/port/win32_msvc. As a last resort, I tried to add it to mingw profile in the hope that there is some support for int64 there, but it does not have any effect.

The second question is: why mvn uses the msvc as default build profile? The answer might be in

<activation>
    <property>
        <name>env.VCINSTALLDIR</name>
    </property>
</activation>

as opposed to

<activation>
    <property>
        <name>env.MSYSTEM</name>
        <value>MINGW64</value>
    </property>
</activation>

...but I could not find any of such system variables (I might be wrong here, this is my first encounter with POM structure at this level).

#7 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

Please find the attached logs from
[...]
and
[...]

I noticed that the msvc profile has an additional include path: ${PGSQL_INCLUDEDIR-SERVER}/port/win32_msvc. As a last resort, I tried to add it to mingw profile in the hope that there is some support for int64 there, but it does not have any effect.

The second question is: why mvn uses the msvc as default build profile? The answer might be in
[...]
as opposed to

[...]

...but I could not find any of such system variables (I might be wrong here, this is my first encounter with POM structure at this level).

Try to add system variable:
MSYSTEM to be MINGW64

What version of the Maven do you use?

#8 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Try to add system variable:
MSYSTEM to be MINGW64

I will try that.

What version of the Maven do you use?

Apache Maven 3.6.0 and gcc (GCC) 8.2.0.

#9 Updated by Ovidiu Maxiniuc over 5 years ago

After I

set MSYSTEM=MINGW64

the mingw profile is indeed used. However, the issues remain.

#10 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

After I
[...]
the mingw profile is indeed used. However, the issues remain.

Take a look at your .m2\repository\com\git-hub\maven-nar\nar-maven-plugin\version directory. The version is 3.2.3 in my system, you can have another number. There is a file nar-maven-plugin-version.jar inside directory.

This is module used to build native code and is failing now. Take inside the jar(easy in Ubuntu), go to the com/git-hub/maven-nar tree inside jar. You will see the aol.properties file. Extract it temporary and review, may be you have to patch this config file.

The aol.properties has a set of the compiler/linker references to instruct maven what to use with compile/link and options for compile/link. If the file has wrong values - the native module will not be built. I have attached the sample file for you to compare. I guess the lines in file are pretty clear to see the meaning.

If you will patch the file - the name(aol.properties) and the location inside jar must be exactly the same.

#12 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Take a look at your .m2\repository\com\git-hub\maven-nar\nar-maven-plugin\version directory. The version is 3.2.3 in my system, you can have another number. There is a file nar-maven-plugin-version.jar inside directory.

I have this jar: .m2\repository\com\github\maven-nar\nar-maven-plugin\3.2.3\nar-maven-plugin-3.2.3.jar (note the path is a bit different, but this is not relevant).

If you will patch the file - the name(aol.properties) and the location inside jar must be exactly the same.

I did that. The log did changed a bit, but the problems are still there.

I tracked the source of the error message (#error must have a working 64-bit integer datatype) and I got to c:\Program Files\PostgreSQL\9.5\include\server\c.h. At lines 328-356 I can read (I added indentation so that it is more visible):

/*
 * 64-bit integers
 */
#ifdef HAVE_LONG_INT_64
    /* Plain "long int" fits, use it */

    #ifndef HAVE_INT64
        typedef long int int64;
    #endif
    #ifndef HAVE_UINT64
        typedef unsigned long int uint64;
    #endif
    #define INT64CONST(x)  (x##L)
    #define UINT64CONST(x) (x##UL)
#elif defined(HAVE_LONG_LONG_INT_64)
    /* We have working support for "long long int", use that */

    #ifndef HAVE_INT64
        typedef long long int int64;
    #endif
    #ifndef HAVE_UINT64
        typedef unsigned long long int uint64;
    #endif
    #define INT64CONST(x)  (x##LL)
    #define UINT64CONST(x) (x##ULL)
#else
    /* neither HAVE_LONG_INT_64 nor HAVE_LONG_LONG_INT_64 */
    #error must have a working 64-bit integer datatype
#endif

I did some digging and HAVE_LONG_LONG_INT_64 macro that is conditionally defined in c:\Program Files\PostgreSQL\9.5\include\server\pg_config.h like this:

/* Define to 1 if `long long int' works and is 64 bits. */
#if (_MSC_VER > 1200)
#define HAVE_LONG_LONG_INT_64 1
#endif

IIRC, MSC_VER is defined as 1200 in Visual Studio 6.0. And most likely not defined at all in g++.

The another macro HAVE_LONG_INT_64 is used as a fallback, but I could not find its definition anywhere.

#13 Updated by Eugenie Lyzenko over 5 years ago

As far as I know the MinGW64 has support for long long int. it is __int64. So what you need to do is to try to define:
-DHAVE_LONG_LONG_INT_64 and -Dint64=__int64' for CPPFLAGS related entries in aol.properties you are going to use.

My understanding is in MinGW the unsigned __int64 is a UINT64.

#14 Updated by Ovidiu Maxiniuc over 5 years ago

I finally get it 100% compiled. The problem is that psql does not know what are the 'resources' (read 'header declarations') of the compiler comes with. The .h files check some flags and if not set then psql API adds its own declarations. I finally used the following line in aol.properties:

amd64.Windows.gpp.c.defines=Windows HAVE_LONG_LONG_INT_64 HAVE_GETTIMEOFDAY HAVE_STRUCT_SOCKADDR_STORAGE HAVE_STRUCT_SOCKADDR_STORAGE_SS_FAMILY PG_PRINTF_ATTRIBUTE=gnu_printf

#15 Updated by Ovidiu Maxiniuc over 5 years ago

I think I was a little to enthusiastic about the successful build. Apparently there is an issue with the binaries because the extension cannot be created. After further investigations I found these lines in a log file:

LOG:  server process (PID 6028) was terminated by exception 0xC0000005
DETAIL:  Failed process was running: create extension pljava
HINT:  See C include file "ntstatus.h" for a description of the hexadecimal value.
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
LOG:  all server processes terminated; reinitializing

I could not find any ntstatus.h anywhere on my HDD, but the error code 0xC0000005 seems to be quite known. A look in MSDN (https://msdn.microsoft.com/en-us/library/cc704588.aspx) tells that it's STATUS_ACCESS_VIOLATION. This looks bad because I don't think we can what happens behind the scene, when pljava-so-1.5.0.dll is loaded and what is wrong with it.

#16 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

I think I was a little to enthusiastic about the successful build. Apparently there is an issue with the binaries because the extension cannot be created. After further investigations I found these lines in a log file:
[...]

I could not find any ntstatus.h anywhere on my HDD, but the error code 0xC0000005 seems to be quite known. A look in MSDN (https://msdn.microsoft.com/en-us/library/cc704588.aspx) tells that it's STATUS_ACCESS_VIOLATION. This looks bad because I don't think we can what happens behind the scene, when pljava-so-1.5.0.dll is loaded and what is wrong with it.

Yes, this is the real issue. Please check:
1. If the DLL binary bitness matches the OS and other DB components that is working with. I mean bitness of the rest DB binaries.
2. Did you set up MinGW compile/link flags properly. Comparing to P2J.DLL makefile options.
3. Whether the pljava-so-1.5.0.dll uses mixed MSVC and MinGW codes together. Especially on the DLL init stage.
4. May be it makes sense to look into C native code to find out some code that can be incompatible with MinGW or currently used OS. For example some Windows subsystem can be really 32-bit only even with 64-bit of the OS.

Also it would be great to have more debugging info for 0xC0000005. The module name and address will help significantly. What the Windows Error logger tells for exception details?

#17 Updated by Eric Faulhaber over 5 years ago

https://tada.github.io/pljava/build/mingw64.html implies that the PostgreSQL server needs to have been built using mingw, in order to build PL/Java with mingw. Is that the case here?

#18 Updated by Ovidiu Maxiniuc over 5 years ago

Also it would be great to have more debugging info for 0xC0000005. The module name and address will help significantly. What the Windows Error logger tells for exception details?

I don't have the memory address that caused the exception. The Windows' Event Viewer reports only the messages I wrote above.

#19 Updated by Ovidiu Maxiniuc over 5 years ago

Eric Faulhaber wrote:

https://tada.github.io/pljava/build/mingw64.html implies that the PostgreSQL server needs to have been built using mingw, in order to build PL/Java with mingw. Is that the case here?

Yes. Compiling with mingw is optional I think, but this is the way we do it - as opposed to using msvc complier/linker.

#20 Updated by Ovidiu Maxiniuc over 5 years ago

I tried today with a fresh new system, hoping that the initial system could have been somehow corrupted. There is no change. I encountered the exact behaviour.

Eugenie Lyzenko wrote:

Yes, this is the real issue. Please check:
1. If the DLL binary bitness matches the OS and other DB components that is working with. I mean bitness of the rest DB binaries.

I inspected the native built library, PLJAVA-SO-1.5.0.DLL. It only depends on KERNEL32.DLL, MSVCRT.DLL and POSTGRES.EXE. All of them are 64-bit binaries.

2. Did you set up MinGW compile/link flags properly. Comparing to P2J.DLL makefile options.

P2J.DLL depends on a few more DLL files. All are 64-bit. I don't recall using this file. I only used the server side of FWD when working on Windows.

3. Whether the pljava-so-1.5.0.dll uses mixed MSVC and MinGW codes together. Especially on the DLL init stage.

I cannot tell that, but which MSVC code? I do not have any installed. Except for the standard redistributable that were installed automatically. Note that the machine has SQL Server 2012 up & running.

4. May be it makes sense to look into C native code to find out some code that can be incompatible with MinGW or currently used OS. For example some Windows subsystem can be really 32-bit only even with 64-bit of the OS.

based on my dependency investigations I think we can exclude any 32 bit issues.

I re-read some articles on web and found this:

If your PostgreSQL server was built using MinGW-w64 (which you can check by running pg_config and looking for mingw), then you can build PL/Java using the same toolchain.

This is strange. As I understand this now, this is a possible cause why PL/Java compiled with MinGW? Apparently my PostgreSQL is not built with MinGW. I do not know what was used for building PostgreSQL I downloaded and installed.
However, I tried to find such a binary, of PostgreSQL built with MinGW. I could only find something like PostgreSQL 9.3 but I really do not trust the sources.

#21 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

I tried today with a fresh new system, hoping that the initial system could have been somehow corrupted. There is no change. I encountered the exact behaviour.

Eugenie Lyzenko wrote:

Yes, this is the real issue. Please check:
1. If the DLL binary bitness matches the OS and other DB components that is working with. I mean bitness of the rest DB binaries.

I inspected the native built library, PLJAVA-SO-1.5.0.DLL. It only depends on KERNEL32.DLL, MSVCRT.DLL and POSTGRES.EXE. All of them are 64-bit binaries.

2. Did you set up MinGW compile/link flags properly. Comparing to P2J.DLL makefile options.

P2J.DLL depends on a few more DLL files. All are 64-bit. I don't recall using this file. I only used the server side of FWD when working on Windows.

Did you use MinGW options -Wl,--subsystem,windows,--kill-at while building PL/Java module?

#22 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

If your PostgreSQL server was built using MinGW-w64 (which you can check by running pg_config and looking for mingw), then you can build PL/Java using the same toolchain.

This is strange. As I understand this now, this is a possible cause why PL/Java compiled with MinGW? Apparently my PostgreSQL is not built with MinGW. I do not know what was used for building PostgreSQL I downloaded and installed.
However, I tried to find such a binary, of PostgreSQL built with MinGW. I could only find something like PostgreSQL 9.3 but I really do not trust the sources.

Do you mean this build: https://sourceforge.net/projects/postgresql-mingw-w64/ ?

#23 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Do you mean this build: https://sourceforge.net/projects/postgresql-mingw-w64/ ?

I downloaded the package at location. There are two archives inside - for 32 and 64 bit. But they only contain some plugin libraries:
libcurl-4.dll, libiconv-2.dll, liblber.dll, libldap.dll, libmysqld.dll, libsqlite3-0.dll, libxml2-2.dll, libyajl.dll, oci.dll with their support bundles (*.control, *.sql, etc).
So this is not ProgreSQL server.

#24 Updated by Ovidiu Maxiniuc over 5 years ago

I found an installed a PostgreSQL that is apparently built with MinGW. It is available here: https://www.openscg.com/bigsql/postgresql/installers.jsp/, see "PostgreSQL 9.5.15" frame. It installs a bit different, in a different location and with a bit different directory structure. The pg_config reports the following:

BINDIR = C:/Apps/POSTGR~1/pg95/bin
DOCDIR = C:/Apps/POSTGR~1/pg95/share/doc/POSTGR~1
HTMLDIR = C:/Apps/POSTGR~1/pg95/share/doc/POSTGR~1
INCLUDEDIR = C:/Apps/POSTGR~1/pg95/include
PKGINCLUDEDIR = C:/Apps/POSTGR~1/pg95/include/POSTGR~1
INCLUDEDIR-SERVER = C:/Apps/POSTGR~1/pg95/include/POSTGR~1/server
LIBDIR = C:/Apps/POSTGR~1/pg95/lib
PKGLIBDIR = C:/Apps/POSTGR~1/pg95/lib/POSTGR~1
LOCALEDIR = C:/Apps/POSTGR~1/pg95/share/locale
MANDIR = C:/Apps/PostgreSQL/pg95/share/man
SHAREDIR = C:/Apps/POSTGR~1/pg95/share/POSTGR~1
SYSCONFDIR = C:/Apps/PostgreSQL/pg95/etc/postgresql
PGXS = C:/Apps/POSTGR~1/pg95/lib/POSTGR~1/pgxs/src/MAKEFI~1/pgxs.mk
CONFIGURE = '--prefix=C:/msys64/opt/pgbin-build/pgbin/bin/../20181106_0039/build/pg95-9.5.15-1-win64' '--enable-integer-datetimes' '--enable-thread-safety' '--with-libxml' '--with-libxslt' '--with-ossp-uuid' '--enable-nls' '--with-openssl' '--with-ldap' '--with-python' '--with-perl' '--with-tcl' '--with-libraries=/opt/pgbin-build/pgbin/shared/win64/lib:/usr/local/lib:/usr/local/ssl/lib' '--with-includes=/usr/local/include:/usr/local/ssl/include' 'CFLAGS=-O2 -DMS_WIN64 -I/opt/pgbin-build/pgbin/shared/win64/include'
CC = gcc
CPPFLAGS = -I./src/include/port/win32 -DEXEC_BACKEND -I/usr/local/include/libxml2 -I/usr/local/include -I/usr/local/ssl/include -I../../../src/include/port/win32
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -DMS_WIN64 -I/opt/pgbin-build/pgbin/shared/win64/include
CFLAGS_SL =
LDFLAGS = -Wl,--allow-multiple-definition -Wl,--disable-auto-import -L/usr/local/lib -LC:/msys64/mingw64/lib -L/opt/pgbin-build/pgbin/shared/win64/lib -L/usr/local/lib -L/usr/local/ssl/lib -Wl,--as-needed
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lintl -lxslt -lxml2 -lssl -lcrypto -lz -lws2_32 -lm  -lws2_32
VERSION = PostgreSQL 9.5.15                                                                                                                                              

Notice -LC:/msys64/mingw64/lib in LDFLAGS. There are some problems here:
  • the paths do not match the paths from my system. For example, the -L directive above should have read -LC:/Apps/msys64/mingw64/lib.
  • some paths are UNIX-like.

I tried to adjust the paths for a closer match and started maven. It stops with the following error:

[ERROR] C:\Apps\POSTGR~1\pg95\include\POSTGR~1\server/c.h:99:10: fatal error: libintl.h: No such file or directory
[ERROR]  #include <libintl.h>
[ERROR]           ^~~~~~~~~~~
[ERROR]                  from C:\Apps\pljava-master\pljava-so\src\main\include/pljava/PgObject.h:12,
[ERROR] compilation terminated.                                                                                                                                         

In fact it repeats itself for different files (20-30 times).

I do not have such file libintl.h anywhere on my drive. Apparently, it is missing from PostgreSQL's include or it was available when it was built. I think this PostgreSQL distribution was cross-compiled.

#25 Updated by Greg Shah over 5 years ago

I suspect you will need to build the project yourself from source. In this way you can ensure that everything is matching.

#26 Updated by Greg Shah over 5 years ago

Ovidiu: Are you close to resolving this? If not, I propose that Eugenie can take the next steps. He is quite a wizard with native code issues. :)

#27 Updated by Ovidiu Maxiniuc over 5 years ago

Greg Shah wrote:

Ovidiu: Are you close to resolving this? If not, I propose that Eugenie can take the next steps. He is quite a wizard with native code issues. :)

I agree. If Eugenie can solve the build/install of PL/Java, it would be great. I do not have many ideas. I was about to attempt to compile PL/Java with MSVC compiler. Just to see it works. There was the free Visual Studio Express, but now they offer Visual Studio Community (https://visualstudio.microsoft.com/free-developer-offers/) which is advertised as free.

Anyway, I an thinking whether we should offer the PL/Java package already pre-build (I believe we do similar for libffi) because of these issues.

After that, I will continue updating the https://proj.goldencode.com/projects/p2j/wiki/Database_Server_Setup_for_PostgreSQL_on_Windows with the rest of instructions. It is completed up to Load FWD's User Defined Function Library. The ending part is not in final form.

#28 Updated by Greg Shah over 5 years ago

I was about to attempt to compile PL/Java with MSVC compiler. Just to see it works. There was the free Visual Studio Express, but now they offer Visual Studio Community (https://visualstudio.microsoft.com/free-developer-offers/) which is advertised as free.

I think this is a good idea. Eugenie is unavailable until the end of the week, so please do try this compile to see if it resolves the issue.

If it doesn't work, then Eugenie can take this when he gets back, unless you have other ideas that can be checked quickly.

Anyway, I an thinking whether we should offer the PL/Java package already pre-build (I believe we do similar for libffi) because of these issues.

Agreed.

#29 Updated by Ovidiu Maxiniuc over 5 years ago

I installed Visual Studio 2017 Community edition. I cannot build. Maven keeps complaining that Could not launch cmd.exe /X /C "gcc --version".

Of course, I 'deactivated' gcc by altering its path so the OS will not find it. Otherwise Maven will build with success.

I tried to force Maven to use the newly installed msvc:
  • dropped the MSYSTEM environment variable;
  • added VCINSTALLDIR=c:\Progra~2\MIB055~1\2017\Community\VC (see note-6 above for the reason of these two);
  • added the path to cl compiler and link linker (it looks like C:\PROGRA~2\MIB055~1\2017\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64 on my system) to environment's PATH variable;
  • force msvc build profile in Maven command line: mvn clean install -P compiler-msvc;
  • cleaned Maven cache (by removing .m2 directory from home dir).
  • altered aol.properties to update/add following lines:
    x86.Windows.msvc.linker.options=/LIBPATH:C:\\PROGRA~2\\WI3CF2~1\\10\\Lib\\10.0.17763.0\\ucrt\\x86 /LIBPATH:C:\\PROGRA~2\\WI3CF2~1\\10\\Lib\\10.0.17763.0\\um\\x86 /LIBPATH:C:\\PROGRA~2\\MIB055~1\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\lib\\x86 
    x86.Windows.msvc.c.options=-IC:\\PROGRA~2\\MIB055~1\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\include -IC:\\PROGRA~2\\WI3CF2~1\\10\\Include\\10.0.17763.0\\ucrt -IC:\\PROGRA~2\\WI3CF2~1\\10\\Include\\10.0.17763.0\\um -IC:\\PROGRA~2\\WI3CF2~1\\10\\Include\\10.0.17763.0\\shared 
    amd64.Windows.msvc.linker.options=/LIBPATH:C:\\PROGRA~2\\WI3CF2~1\\10\\Lib\\10.0.17763.0\\ucrt\\x64 /LIBPATH:C:\\PROGRA~2\\WI3CF2~1\\10\\Lib\\10.0.17763.0\\um\\x64 /LIBPATH:C:\\PROGRA~2\\MIB055~1\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\lib\\x64 
    amd64.Windows.msvc.c.options=-IC:\\PROGRA~2\\MIB055~1\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\include -IC:\\PROGRA~2\\WI3CF2~1\\10\\Include\\10.0.17763.0\\ucrt -IC:\\PROGRA~2\\WI3CF2~1\\10\\Include\\10.0.17763.0\\um -IC:\\PROGRA~2\\WI3CF2~1\\10\\Include\\10.0.17763.0\\shared
    

Eventually, the build was successful:

[INFO] Reactor Summary for PostgreSQL PL/Java 1.5.0:
[INFO] 
[INFO] PostgreSQL PL/Java ................................. SUCCESS [  1.203 s]
[INFO] PL/Java API ........................................ SUCCESS [  2.125 s]
[INFO] PL/Java backend Java code .......................... SUCCESS [  2.438 s]
[INFO] PL/Java backend native code ........................ SUCCESS [ 41.041 s]
[INFO] PL/Java Deploy ..................................... SUCCESS [  6.250 s]
[INFO] PL/Java Ant tasks .................................. SUCCESS [ 13.468 s]
[INFO] PL/Java examples ................................... SUCCESS [  1.422 s]
[INFO] PL/Java packaging .................................. SUCCESS [  3.375 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Generating pljava-1_5_0-msvc\pljava-packaging\target\pljava-pg9.5-amd64-Windows-msvc.jar.

Unfortunately, after the extension is installed, the database server is reset again. I can see this in pg_log:

2018-12-03 23:20:53 EET LOG:  database system is ready to accept connections
2018-12-03 23:23:33 EET LOG:  server process (PID 2052) was terminated by exception 0xC0000005
2018-12-03 23:23:33 EET DETAIL:  Failed process was running: CREATE EXTENSION pljava
2018-12-03 23:23:33 EET HINT:  See C include file "ntstatus.h" for a description of the hexadecimal value.
2018-12-03 23:23:33 EET LOG:  terminating any other active server processes
2018-12-03 23:23:33 EET WARNING:  terminating connection because of crash of another server process
2018-12-03 23:23:33 EET DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2018-12-03 23:23:33 EET HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2018-12-03 23:23:33 EET WARNING:  terminating connection because of crash of another server process
2018-12-03 23:23:33 EET DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2018-12-03 23:23:33 EET HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2018-12-03 23:23:33 EET LOG:  all server processes terminated; reinitializing

#30 Updated by Greg Shah over 5 years ago

  • Assignee changed from Ovidiu Maxiniuc to Eugenie Lyzenko

OK, thanks for the update.

Eugenie will take this when he gets back.

#31 Updated by Eugenie Lyzenko over 5 years ago

My started config environment:
1. PostgreSQL 9.5.15 downloaded as binary (to be on the almost same level as in Linux)
2. GCC version 4.9.0 (mingw64)
3. Oracle JDK 1.8.0_192
4. PLJava 1.5.2 source zip

The aol.properties with changes described in note #3759-14. The building fails with errors below(failure on linking stage).

Ovidiu,

Have you encountered this issue with your build? Do you have MSYS installed package? The command mvn clean install was executed from MSYS shell or from regular command prompt?

Native compilation issue log:

...
[ERROR] C:\opt\pljava_152\pljava-so\src\main\c\SubXactListener.c: In function 'subXactCB':
[WARNING] C:\opt\pljava_152\pljava-so\src\main\c\SubXactListener.c:26:2: warning: enumeration value 'SUBXACT_EVENT_PRE_COMMIT_SUB' not handled in switch [-Wswitch]
[ERROR]   switch(event)
[ERROR]   ^
[ERROR] C:\opt\pljava_152\pljava-so\src\main\c\XactListener.c: In function 'xactCB':
[WARNING] C:\opt\pljava_152\pljava-so\src\main\c\XactListener.c:25:2: warning: enumeration value 'XACT_EVENT_PARALLEL_COMMIT' not handled in switch [-Wswitch]
[ERROR]   switch(event)
[ERROR]   ^
[WARNING] C:\opt\pljava_152\pljava-so\src\main\c\XactListener.c:25:2: warning: enumeration value 'XACT_EVENT_PARALLEL_ABORT' not handled in switch [-Wswitch]
[WARNING] C:\opt\pljava_152\pljava-so\src\main\c\XactListener.c:25:2: warning: enumeration value 'XACT_EVENT_PRE_COMMIT' not handled in switch [-Wswitch]
[WARNING] C:\opt\pljava_152\pljava-so\src\main\c\XactListener.c:25:2: warning: enumeration value 'XACT_EVENT_PARALLEL_PRE_COMMIT' not handled in switch [-Wswitch]
[WARNING] C:\opt\pljava_152\pljava-so\src\main\c\XactListener.c:25:2: warning: enumeration value 'XACT_EVENT_PRE_PREPARE' not handled in switch [-Wswitch]
[INFO] 53 files were compiled.
[INFO] 53 files were compiled.
[INFO] Linking...
[INFO] Linking...
[INFO] Starting link {4.9.0 -Wl,--export-all-symbols -m64 -Wl,--subsystem,windows,--kill-at -shared -LC:\POSTGR~1\9.5\lib -Bdynamic -lpostgres -shared-libgcc}
[INFO] Starting link {4.9.0 -Wl,--export-all-symbols -m64 -Wl,--subsystem,windows,--kill-at -shared -LC:\POSTGR~1\9.5\lib -Bdynamic -lpostgres -shared-libgcc}
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x131): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x13f): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x166): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x189): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x197): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x1b1): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x1bf): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x210): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x21e): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x245): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x268): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x276): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x290): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x29e): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x2ff): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x30d): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x338): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x445): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x453): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x46d): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x47b): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x4cd): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x4db): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x502): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x539): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x547): undefined reference to `__imp_error_context_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x561): undefined reference to `__imp_PG_exception_stack'
[ERROR] C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o:AclId.c:(.text+0x56f): undefined reference to `__imp_error_context_stack'
[ERROR] c:/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.9.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\opt\pljava_152\pljava-so\target\nar\obj\amd64-Windows-gpp\AclId.o: bad reloc address 0x0 in section `.data'
[ERROR] collect2.exe: error: ld returned 1 exit status
...

#32 Updated by Eugenie Lyzenko over 5 years ago

Looks like I have worked PLJava extension. What was changed:
0. MSYSTEM must be MINGW64
1. Upgrade MAVEN to 3.6.0
2. Install PostgreSQL 9.5.15. This is the MinGW based build from https://www.openscg.com/bigsql/postgresql/installers.jsp/.
3. Use aol.mingw.properties provided here.
4. Install libintl.h into MinGW64 include dir provided here(actually I guess this is just make compiler to be happy).
5. Install openssl header file directory into MinGW64 include dir. Need to copy full directory from MSVC based PostgreSQL build. This is the serious issue of the MinGW based build and I have no explanation why these files are missing there. This looks like "almost complete package" approach allowing to use PostgreSQL as application but stopping to use it for development.
6. Run mvn clean install -Dnar.aolProperties=aol.mingw.properties to build extension. Was OK on my system. The command must be run in regular shell, not in VC environment.
7. Install extension by java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar
8. Final steps:
- psql
- SET pljava.libjvm_location TO 'C:\some\pathh\...\jvm.dll';
- CREATE EXTENSION pljava;
- GRANT USAGE ON LANGUAGE java TO PUBLIC;
- restart the server to make sure it works.

Ovidiu, please confirm if this scenario works on your system. Or please let me know what approach did you use to verify the new extension is working.

My conclusion is the main issue is PostgreSQL and PLJava must be build using the same compiler(MinGW in our case).

I have no PLJava build for MSVC based PostgreSQL so I still do not know why this combination does not work. May be we need to investigate this case additionally. Continue working here. During several PostgreSQL installations(now I have 3 different DB servers running, 9.4, 9.5(MS), 9.5(MinGW)) the different MS runtimes was also installed so need to investigate if the build can work without them.

#33 Updated by Eugenie Lyzenko over 5 years ago

One addition. The system used is Windows 7 64-bit Ultimate.

#34 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Looks like I have worked PLJava extension. What was changed:
0. MSYSTEM must be MINGW64
1. Upgrade MAVEN to 3.6.0
2. Install PostgreSQL 9.5.15. This is the MinGW based build from https://www.openscg.com/bigsql/postgresql/installers.jsp/.
3. Use aol.mingw.properties provided here.

I tried with all above. The aol.mingw.properties extracts only the mingw -related directives from packaged aol.properties. The only difference is that I used a much newer gcc compiler:

gcc --version
gcc (Rev3, Built by MSYS2 project) 8.2.0

I used both msys2 and standard console. Indeed, building from msys2 console was not successful.

4. Install libintl.h into MinGW64 include dir provided here(actually I guess this is just make compiler to be happy).

I stopped here see end of note 24 above. Where did you get libintl.h? It is legal to distribute the file alone? In fact this was the reason my build with PostgreSQL 9.5.15 (from https://www.openscg.com/bigsql/postgresql/installers.jsp/) was not successful.

5. Install openssl header file directory into MinGW64 include dir. Need to copy full directory from MSVC based PostgreSQL build. This is the serious issue of the MinGW based build and I have no explanation why these files are missing there. This looks like "almost complete package" approach allowing to use PostgreSQL as application but stopping to use it for development.

I did not reach this point when building mingw version of PSQL. My successful builds using mingw and msvc of MSVC based PostgreSQL did not require this.

6. Run mvn clean install -Dnar.aolProperties=aol.mingw.properties to build extension. Was OK on my system. The command must be run in regular shell, not in VC environment.
7. Install extension by java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar

This requires special rights. I used Administrator console for successful builds.

8. Final steps:
- psql
- SET pljava.libjvm_location TO 'C:\some\pathh\...\jvm.dll';

I was not aware of this setting in psql interface. I have this instead added to postgresql.conf:

pljava.libjvm_location = 'C:\PROGRA~1\Java\JDK18~1.0_1\jre\bin\server\jvm.dll'

- CREATE EXTENSION pljava;

This is where my build binaries failed:

postgres=# CREATE EXTENSION pljava;
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>

- GRANT USAGE ON LANGUAGE java TO PUBLIC;
- restart the server to make sure it works.

Ovidiu, please confirm if this scenario works on your system. Or please let me know what approach did you use to verify the new extension is working.

I need to cleanup and rebuild using your instructions. I will be back in about 1 h with the result.

My conclusion is the main issue is PostgreSQL and PLJava must be build using the same compiler(MinGW in our case).

I also tried with msvc compiler (both PostgreSQL and PLJava) and I got same exception. What is strange is that I get the same crash for 3 different PC (two at the office and my personal laptop at home).

I have no PLJava build for MSVC based PostgreSQL so I still do not know why this combination does not work. May be we need to investigate this case additionally. Continue working here. During several PostgreSQL installations(now I have 3 different DB servers running, 9.4, 9.5(MS), 9.5(MinGW)) the different MS runtimes was also installed so need to investigate if the build can work without them.

#35 Updated by Ovidiu Maxiniuc over 5 years ago

Yes, this time the extension was created with success. I also upgraded to 1.5.2 (I used 1.5.0 previously). I do not know if this made any difference.
The only issues I encountered were at step 5. Beside openssl (I misunderstood at first and downloaded and installed true openssl), the full server\port\win32\ directory must also be copied to include directory of mingw. Most of these files only contains commented code but they are needed by compiler.

Indeed, I find two strange things:
  • that this mingw version of PostgerSQL does not come with the openssl directory
  • why win32 was not recognized. If I look into pq_config output I can clearly see:
    INCLUDEDIR-SERVER = C:/Apps/POSTGR~1/pg95/include/POSTGR~1/server
    CPPFLAGS = -I./src/include/port/win32 -DEXEC_BACKEND -I/usr/local/include/libxml2 -I/usr/local/include -I/usr/local/ssl/include -I../../../src/include/port/win32
    

    The path to PostgreSQL is correct, but the paths to desired includes seem broken. The actual absolute path is:
    c:\Apps\PostgreSQL\pg95\include\postgresql\server\port\win32\

#36 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:
...

3. Use aol.mingw.properties provided here.

I tried with all above. The aol.mingw.properties extracts only the mingw -related directives from packaged aol.properties.

This allows to use it in command line leaving original copy in maven jar untouched.

The only difference is that I used a much newer gcc compiler:
[...]

Why do you need the compiler other than we provide for FWD package? The 4.9.0 is known as working but I can not tell the same for other versions(including newer one) because if I recall we had an issues with some of them(especially with Windows console client).

Do we have a plans to upgrade our prepackaged versions.

4. Install libintl.h into MinGW64 include dir provided here(actually I guess this is just make compiler to be happy).

I stopped here see end of note 24 above. Where did you get libintl.h? It is legal to distribute the file alone? In fact this was the reason my build with PostgreSQL 9.5.15 (from https://www.openscg.com/bigsql/postgresql/installers.jsp/) was not successful.

I guess its is legal: https://github.com/git-for-windows/git-sdk-64/blob/master/mingw64/include/libintl.h.

I have no PLJava build for MSVC based PostgreSQL so I still do not know why this combination does not work. May be we need to investigate this case additionally. Continue working here. During several PostgreSQL installations(now I have 3 different DB servers running, 9.4, 9.5(MS), 9.5(MinGW)) the different MS runtimes was also installed so need to investigate if the build can work without them.

The question remaining unclear for me - how can I get the MSVC based PostgreSQL and Mingw64 based compiled PLJava combination?

#37 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Ovidiu Maxiniuc wrote:
This allows to use it in command line leaving original copy in maven jar untouched.

That's cool.

Why do you need the compiler other than we provide for FWD package? The 4.9.0 is known as working but I can not tell the same for other versions(including newer one) because if I recall we had an issues with some of them(especially with Windows console client).

I didn't do it on purpose. I just installed msys2 and when updating, pacman installed the newer gcc and make.

Do we have a plans to upgrade our prepackaged versions.

Yes, Greg agreed on this (see note 28, above).

I guess its is legal: https://github.com/git-for-windows/git-sdk-64/blob/master/mingw64/include/libintl.h.

OK.

The question remaining unclear for me - how can I get the MSVC based PostgreSQL and Mingw64 based compiled PLJava combination?

I do not know yet. The interesting thing is that PL/Java built with MSVC Community Edition is also failing to create the extension for me. Now, I have an idea that I would like to test: I have both PostgreSQL-s installed. Do you have any idea how can I alter
java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar
so that the extension go to MSVC instance?
I know that this is a long-shot, but I would give it a try.

#38 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

The question remaining unclear for me - how can I get the MSVC based PostgreSQL and Mingw64 based compiled PLJava combination?

I do not know yet. The interesting thing is that PL/Java built with MSVC Community Edition is also failing to create the extension for me. Now, I have an idea that I would like to test: I have both PostgreSQL-s installed. Do you have any idea how can I alter
java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar
so that the extension go to MSVC instance?

I think it is not possible via changing jar internals. I see only class that can be responsible for proper directory handling: org/gjt/cuspy/JarX.class. No config files inside, so nothing to adjust to new path. However you can try to change PATH variable in a way the pg_config command to be executed from new location. This can help but I'm not sure.

#39 Updated by Eugenie Lyzenko over 5 years ago

Or as alternative you can try to manually copy the files from your postgresql subdir to new location:
1. complete pljava directory.
2. extension/pljava.control file.

#40 Updated by Ovidiu Maxiniuc over 5 years ago

I have good news.

The pljava 1.5.2 package that I built today installs on both instances of PostgreSQL. Indeed, you need to alter the Windows's PATH environment variable so that target pg_config to be find first. There is a catch: to install on msvc instance you need some Administrator rights while the mingw instances does not need any additional rights. This is because the msvc instance is installed in Program Files system folder.

Why this? I do not know, I only can speculate that there is some kind of issue with pljava 1.5.0.

#41 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

I have good news.

The pljava 1.5.2 package that I built today installs on both instances of PostgreSQL. Indeed, you need to alter the Windows's PATH environment variable so that target pg_config to be find first. There is a catch: to install on msvc instance you need some Administrator rights while the mingw instances does not need any additional rights. This is because the msvc instance is installed in Program Files system folder.

Why this? I do not know, I only can speculate that there is some kind of issue with pljava 1.5.0.

I have another theory. The functions __imp_PG_exception_stack and __imp_error_context_stack the MinGW attempting to link with is missing in MSVC PostgreSQL package. There are no such exports in postgres.lib. The postgres.lib library also uses them so the target is somewhere inside Visual Studio libraries I guess. Even if GCC linker ignores this missing calls resolution on building stage(as it happens in your new GCC version) - it can just postpone the trouble to the execution stage. Just because the native code still needs the functions.

#42 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

...

Do we have a plans to upgrade our prepackaged versions.

Yes, Greg agreed on this (see note 28, above).

Sorry, I would have to be more clear. This was a question. Do we really have a plans to upgrade MinGW prebuilt package we provide to configure developer workstation? I do not see any related in #3759-28 above.

This is serious point because in the case we want to upgrade we need to get complete MinGW sources, recompile it in some Windows environment with MSYS and package only MinGW. As far as I know we do not provide MSYS prebuilt binaries, so user needs only ready to use MinGW package. Moreover we will need to retest all our native programs/modules that are built with MinGW. And I think rebuild libffi as well.

In general if all of us will use different versions of GCC it makes debugging pretty problematic due to different results on different systems. Moreover newer version does not automatically mean the better one.

#43 Updated by Greg Shah over 5 years ago

I am open to changing our packaged version if needed to support all functionality. What do you recommend?

We need to make it reasonable to get things like this built. We can also publish our Windows binaries for PL/Java, right?

#44 Updated by Eugenie Lyzenko over 5 years ago

The addition. My previous note about MinGW was wrong. We do not build MinGW itself. We use already compiled binaries from SourceForge. Currently the latest build is from 20180726. I'm not sure what the GCC version is.

#45 Updated by Eugenie Lyzenko over 5 years ago

Greg Shah wrote:

...
We need to make it reasonable to get things like this built. We can also publish our Windows binaries for PL/Java, right?

Yes, we can publish our Windows binaries for PL/Java. We just need to create and test complete install instruction to replace java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar command from PL/Java site which I think will not work due to different PosgreSQL directory installations on different systems.

#46 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Yes, we can publish our Windows binaries for PL/Java. We just need to create and test complete install instruction to replace java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar command from PL/Java site which I think will not work due to different PostgreSQL directory installations on different systems.

Actually this works. At least it worked for me. I build PL/Java using MinGW and MinGW-built PostgreSQL installed in C:\Apps\ProgreSQL\pg9.5\ and I installed it on MSVC-built PostgreSQL which was installed in standard location, C:\Program Files\PostgreSQL\9.5.

As far as I can tell, the java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar command will use the first accessible pg_config from PATH and install at respective location. Since I temporarily renamed the PostgreSQL used for building and added the bin of the msvc-built PostgreSQL instance to PATH, the jar installed to the later location. I tested and the EXTENSION was installed with success.

#47 Updated by Eugenie Lyzenko over 5 years ago

Greg Shah wrote:

I am open to changing our packaged version if needed to support all functionality. What do you recommend?

To answer this question I need to know what is the version of GCC the recent build provides. Then I would search the differences and new features the new version offers. Then if we find the new features are important for us to use or open some attractive perspectives - I would do very careful regression testing for all native programs/modules we build with GCC. Then if everything is OK - we can merge to new GCC version.

If new version is the same or has no attractive features for us or if there are stability issues with new version - my way will be to keep working with current 4.9.0. This is the known approach "Do not fix anything that works".

#48 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

Eugenie Lyzenko wrote:

Yes, we can publish our Windows binaries for PL/Java. We just need to create and test complete install instruction to replace java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar command from PL/Java site which I think will not work due to different PostgreSQL directory installations on different systems.

Actually this works. At least it worked for me. I build PL/Java using MinGW and MinGW-built PostgreSQL installed in C:\Apps\ProgreSQL\pg9.5\ and I installed it on MSVC-built PostgreSQL which was installed in standard location, C:\Program Files\PostgreSQL\9.5.

As far as I can tell, the java -jar pljava-packaging/target/pljava-pg9.5-amd64-Windows-gpp.jar command will use the first accessible pg_config from PATH and install at respective location. Since I temporarily renamed the PostgreSQL used for building and added the bin of the msvc-built PostgreSQL instance to PATH, the jar installed to the later location. I tested and the EXTENSION was installed with success.

This is a great news! We do not care about manual file copy in user documentation. Just configure proper pg_config location.

#49 Updated by Eugenie Lyzenko over 5 years ago

Well, the new version of GCC from 20180726 is 8.2.0. Need to inspect what the new features we have got in this release.

#50 Updated by Ovidiu Maxiniuc over 5 years ago

Greg, how do we handle the missing header files issue? Evidently, we cannot ask the customers to install msvc-built PostgreSQL just because of the missing files? Will we provide them ?

#51 Updated by Greg Shah over 5 years ago

Ovidiu Maxiniuc wrote:

Greg, how do we handle the missing header files issue? Evidently, we cannot ask the customers to install msvc-built PostgreSQL just because of the missing files? Will we provide them ?

We probably should provide them. I prefer for them to be built into our mingw "distribution". Is that feasible? Are all the missing files safe to include (from a licensing perspective)?

#52 Updated by Ovidiu Maxiniuc over 5 years ago

Greg Shah wrote:

Ovidiu Maxiniuc wrote:

Greg, how do we handle the missing header files issue? Evidently, we cannot ask the customers to install msvc-built PostgreSQL just because of the missing files? Will we provide them ?

We probably should provide them. I prefer for them to be built into our mingw "distribution". Is that feasible? Are all the missing files safe to include (from a licensing perspective)?

Beside the libintl.h which Eugenie attached there are two other directories:
  • openssl: 75 files, most of them have OpenSSL project headers. Some are generated, some do not have headers at all.
  • win32 directory has about a dozen files and sub-directories. Neither have any headers. Most of them do not have any active code at all, the others just chain include a couple of other system headers. The total size of this is 1KiB.

I think we can add them to our mingw "distribution".

#53 Updated by Greg Shah over 5 years ago

If our existing distribution will work with just adding these files, I'm OK with that approach.

#54 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

  • win32 directory has about a dozen files and sub-directories. Neither have any headers. Most of them do not have any active code at all, the others just chain include a couple of other system headers. The total size of this is 1KiB.

I think we can add them to our mingw "distribution".

Strange but I do not need this to be copied in MinGW(even for GCC 8.2.0) to build PL/Java binary.

Your PostgreSQL distribution does not have the SomeRootDrivedir\PostgreSQL\pg95\include\postgresql\server\port\win32\?

#55 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Strange but I do not need this to be copied in MinGW(even for GCC 8.2.0) to build PL/Java binary.
Your PostgreSQL distribution does not have the SomeRootDrivedir\PostgreSQL\pg95\include\postgresql\server\port\win32\?

Indeed, it is strange. I have it but the compiler complained about every single file from that directory. Copying it to mingw include directory fixed all issues. See my note-35 above.

#56 Updated by Eugenie Lyzenko over 5 years ago

The my points about new GCC version update is the following. There are a lot of changes between 4.9.0 to 8.2.0. The biggest one on my guess is shifting from C89 to C11 language specification.

Regarding to our native code usage in Windows:
- many sanity checks have been added to avoid wrong memory access in string operations and memcpy.
- pointer arithmetics is also "sanitized". I think it will be not so easy to use integer value as an object pointers as we do in COM ActiveX module. I expect more warnings.
- the Thread Local Storage support has been added. This is what I wanted to have while implementing COM support. Should speed up small allocations/releases.
- another checks to improve dangerous code identifications on compilation stage.
- improved console readability and messages matching the real issue.

So taking into account the percentage of the native code in whole FWD for now my point is merging can be useful but not urgent. I do not see any magic feature we need right now(correct me if I'm wrong). We can plan the shifting to new version when we have 2-4 days to rebuild all other natives(including libffi) and test for critical regressions the spawn.exe and p2j.dll including ChUI console support.

#57 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

Eugenie Lyzenko wrote:

Strange but I do not need this to be copied in MinGW(even for GCC 8.2.0) to build PL/Java binary.
Your PostgreSQL distribution does not have the SomeRootDrivedir\PostgreSQL\pg95\include\postgresql\server\port\win32\?

Indeed, it is strange. I have it but the compiler complained about every single file from that directory. Copying it to mingw include directory fixed all issues. See my note-35 above.

Can you try to move your PostgreSQL to C:\PostgreSQL\pg95\.... and check if it works?

#58 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie Lyzenko wrote:

Can you try to move your PostgreSQL to C:\PostgreSQL\pg95\.... and check if it works?

This is funny, now it works. I removed the win32 directory, so it does not take the file from there. So we can drop this part. Only openssl and libintl.h remain.

#59 Updated by Greg Shah over 5 years ago

We don't need to update the gcc version. Let's just include the missing files and update our distribution download.

We'll need to confirm that the changes don't break the FWD build.

#60 Updated by Eugenie Lyzenko over 5 years ago

Greg Shah wrote:

We don't need to update the gcc version. Let's just include the missing files and update our distribution download.

We'll need to confirm that the changes don't break the FWD build.

Agreed.

#61 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

Eugenie Lyzenko wrote:

Can you try to move your PostgreSQL to C:\PostgreSQL\pg95\.... and check if it works?

This is funny, now it works.

What to say... This PostgreSQL is an example of how to do not the package to deploy.

#62 Updated by Ovidiu Maxiniuc over 5 years ago

Added openssl header files needed to built PL/Java with mingw-built PostgreSQL.

#63 Updated by Ovidiu Maxiniuc over 5 years ago

I started to build PL/Java on customer system but at the end I stumbled on following issue: the javah is missing from JDK 1.8.0_144. According to Oracle, has been superseded by the -h option added to javac in JDK 8, is deprecated as of JDK and might be removed in a future JDK release (https://docs.oracle.com/javase/9/tools/javah.htm#JSWOR687) but it seems that they seem to already drop it in latest updates of Java8 as I found already installed on the customer's system.

The replacement is the support from javac to do the very same job when launched with -h parameter. However, the tools that build the project - nar maven plugin does not support this. I read that they will add support 'for Java 10' in nar-maven-plugin-3.7.0, but when this will be released is unknown, there is only about a month from the release of 3.6.0 (latest release at this time).

#64 Updated by Greg Shah over 5 years ago

It is OK to install a pre-built (on your system) PL/Java on the customer system.

#65 Updated by Eugenie Lyzenko over 5 years ago

Several conclusions about building PL/Java and modified MinGW include directory with Windows environment:

1. The added libintl.h and openssl header file directory into MinGW do not stop the FWD project native modules compilation. It is safe to use.
2. The building PL/Java module was verified on clean system with only MinGW based PostgreSQL. So we can now say we have repeatable OK result. The absolutely mandatory setup points are to have proper MSYSTEM as MINGW64 and respective custom aol.properties file to use in maven command. No other extra tools(like MSYS) are required.
3. When installing PostgreSQL build the default directory(for program files) should be accepted to avoid include files finding issues.
4. Currently the 64-bit Windows 7 was tested. However I do not expect significant issues in further 64-bit Window versions.
5. 32-bit Windows case was not tested.

#66 Updated by Ovidiu Maxiniuc over 5 years ago

Eugenie,

I guess you refer in note-65 of MinGW-built PostgreSQL. Building with MSVC-built PostgreSQL is much simpler (it is the default profile and has all required include files) but the resulted binary fails at runtime.

Also I can confirm that the procedure works fine for systems running Windows10 and the equivalent server OS-es.

#67 Updated by Eugenie Lyzenko over 5 years ago

Ovidiu Maxiniuc wrote:

Eugenie,

I guess you refer in note-65 of MinGW-built PostgreSQL. Building with MSVC-built PostgreSQL is much simpler (it is the default profile and has all required include files) but the resulted binary fails at runtime.

Yes, I refer to the MinGW-built PostgreSQL. As the only way to get working binaries.

#69 Updated by Ovidiu Maxiniuc over 5 years ago

Greg Shah wrote:

What updates need to be made to the documentation chapters so that it is complete?

https://proj.goldencode.com/projects/p2j/wiki/Patching_and_Building_Customized_3rd_Party_Libraries#PLJava
https://proj.goldencode.com/projects/p2j/wiki/Database_Server_Setup_for_PostgreSQL_on_Windows

I updated Database Server Setup for PostgreSQL on Windows. But I linked the libintl.h, aol.mingw.properties and openssl.zip to the attachment of this thread. I do not know if they are visible from there as this is a private tracker. I think they should be copied in a global location and linked from there.

#70 Updated by Greg Shah over 5 years ago

This thread is public, but I agree that the artifacts should be seen from the other two documents as well.

#71 Updated by Greg Shah over 4 years ago

  • Status changed from WIP to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF