FWD » Core Development » Runtime Infrastructure

In setup a private TERMINFO database there are some hints of how we can make our own customized terminal types available without editing the system's TERMINFO.

The short version:

• The TERMINFO_DIRS environment variable will be read by ncurses. It defines a search path to use for looking up the TERMINFO database.
• We would create our own directory that includes only our customized terminal definitions. This directory would be pre-pended to the TERMINFO_DIRS search paths so that it will be searched first.
• We would need to identify the existing paths that should be searched so they can be appended to the TERMINFO_DIRS variable. This allows the standard/unmodified terminal defs to be found in the system's normal TERMINFO paths.
• This environment variable will need to be set for every interactive ChUI FWD client.

I would expect we could install these in a terminfo/ subdirectory of wherever the p2j.jar exists for the FWD client.

See #6340. We could use the same technique to extract this application-specific TERMINFO database and avoid the install step.

With the completion of #5167, we now have the option of using a statically linked ncurses installation rather than the system-wide installation.

Until this #5568 task is complete, we still need terminfo to be edited system-wide, which will still require root/sudo and apt hooks. Let's resolve this so that we are completely independent of the system-wide, privleged environment.

The problem with this approach is it only works in development environments. In production systems we don't have a copy of the project, just have jar files.

Greg Shah wrote:

The problem with this approach is it only works in development environments. In production systems we don't have a copy of the project, just have jar files.

So there is no p2j directory on production system for application deployed?

Question: will these be needed by batch processes, launched maybe via the directory scheduler (so the spawner is involved)?

Also, can we include these in the p2j.jar, and on launch of FWD client (and before initializing the p2j native library), extract them into a temp folder, and 'force' the TERMINFO_DIRS to this temp folder?

Eugenie Lyzenko wrote:

Greg Shah wrote:

The problem with this approach is it only works in development environments. In production systems we don't have a copy of the project, just have jar files.

So there is no p2j directory on production system for application deployed?

Correct.

Question: will these be needed by batch processes, launched maybe via the directory scheduler (so the spawner is involved)?

Yes

Also, can we include these in the p2j.jar, and on launch of FWD client (and before initializing the p2j native library), extract them into a temp folder, and 'force' the TERMINFO_DIRS to this temp folder?

Hmmm. This could work. We would want to only do this if it wasn't already there.

I have confirmation the TERMINFO_DIRS based approach works for having independent terminfo database. Moreover we have to use it anyway because if the clean system has only static based NCURSES in use(the system wide NCURSES was never patched) the default terminfo for NCURSES(usr/share/terminfo) is empty and vt320 support is missing.

For location of the custom TERMINFO_DIRS how about to put it inside every application project? For example for Hotel ChUI it will be hotel_root/terminfo. I think this is more stable then using permanently created subdir inside temporary storage for every new client that is starting. What do you think?

This is a possibility, though we would probably put it into deploy/ since most of the project doesn't get put on a production system.

The thing I don't like about this is that it gets duplicated in every project. Even non-ChUI projects will need it for appserver, batch and even the server side client stuff (which uses libp2j on the server). That seems like a maintenance problem.

I'd still prefer if we had a patching process that could run out of the jar and which would create that directory when needed.

Greg Shah wrote:

This is a possibility, though we would probably put it into deploy/ since most of the project doesn't get put on a production system.

The thing I don't like about this is that it gets duplicated in every project. Even non-ChUI projects will need it for appserver, batch and even the server side client stuff (which uses libp2j on the server). That seems like a maintenance problem.

I'd still prefer if we had a patching process that could run out of the jar and which would create that directory when needed.

I see. But if we want this preparation step to be started within client start-up script we will certainly need to modify all projects(having ChUI client start script). Is this possible compromised way? Or we need this preparation to be executed inside FWD client start-up Java code to avoid changing any *.sh files?

• for spawned clients, the FWD server is responsible of extracting the files and injecting the JVM argument at the spawner command, so the FWD client already has this set
• for 'manual' launched clients (swing,chui,batch), the FWD client's startup checks for this property (if already set or not), and if not, extracts the files and sets this property at the JVM, so the libp2j.so can see it

Eugenie, can this work?

• just always overwrite them
• if we don't overwrite them, then check if the latest version is there or not

Also, this folder should not be random, should be something static (maybe in the work dir of the FWD server or FWD client), so we don't flood the file system with this. Another point is: whatever FWD extracts will be on the FWD server's OS account, and this may or may not be readable by the FWD client's OS account. So maybe we let the FWD client manage the extraction.

I think the good location is client side, ConsoleHelper.initChui() method:

   public static void initChui()
   {
      initConsole(); // call native function to initialize console.
   }

We can modify environment variable in initConsole() JNI implementation (terminal_linux.c). But before this we need to unpack terminfo from p2j.jar into directory like deploy/lib/terminfo or deploy/terminfo.

The created directory will be passed to initConsole() as option. Passing NULL will mean do not change the environment(Windows case for example). The pseudo-code will be:

   public static void initChui()
   {
      String terminfoDir = setUpCustomTerminfo();
      initConsole(terminfoDir); // call native function to initialize console.
   }

A production environment will not have a deploy/lib or whatever structure. The only folders which can be assumed are 'working dir' (the current JVM process dir) and the OS temporary folder.

Also, we need to ensure that access to that location is 'thread-safe' - multiple clients may start to write at the same time, or update the files, etc.

Constantin Asofiei wrote:

A production environment will not have a deploy/lib or whatever structure. The only folders which can be assumed are 'working dir' (the current JVM process dir) and the OS temporary folder.

Please clarify. In this case what is the directory the libp2j.so will be loaded from?

Eugenie Lyzenko wrote:

Constantin Asofiei wrote:

A production environment will not have a deploy/lib or whatever structure. The only folders which can be assumed are 'working dir' (the current JVM process dir) and the OS temporary folder.

Please clarify. In this case what is the directory the libp2j.so will be loaded from?

Whatever was configured via -Djava.library.path=. The point was to not make assumptions about some directory structure.

Constantin Asofiei wrote:

Eugenie Lyzenko wrote:

Constantin Asofiei wrote:

A production environment will not have a deploy/lib or whatever structure. The only folders which can be assumed are 'working dir' (the current JVM process dir) and the OS temporary folder.

Please clarify. In this case what is the directory the libp2j.so will be loaded from?

Whatever was configured via -Djava.library.path=. The point was to not make assumptions about some directory structure.

OK. The implementation plan is:
1. Include terminfo into FWD repo.
2. Making changes for FWD build.xml script to automatically create terminfo.zip file for FWD/build/lib directory. But only for Linux distribution.
3. When some project calls ant deploy.prepare this zip file is copied into the same directory that has libp2j.so.

These steps above are already implemented and tested.

4. When it is time to load libp2j FWD will check if the terminfo.zip exists in same dir as libp2j.so. If yes the Java code will extract the archive in this directory if not yet done.
5. Then the native code will check if terminfo directory exists and if yes and having the terminfo files changes the environment variable to set override for TERMINFO_DIRS just before new terminal instance is creating.

Eugenie Lyzenko wrote:

OK. The implementation plan is:
1. Include terminfo into FWD repo.
2. Making changes for FWD build.xml script to automatically create terminfo.zip file for FWD/build/lib directory. But only for Linux distribution.

Does the archive target for deploy also include the terminfo.zip?

3. When some project calls ant deploy.prepare this zip file is copied into the same directory that has libp2j.so.

Has ant deployapp.prepare been tested? This does not perform a copy of the FWD libraries.

These steps above are already implemented and tested.

Is there a branch where these are available?

Roger Borrello wrote:

Eugenie Lyzenko wrote:

OK. The implementation plan is:
1. Include terminfo into FWD repo.
2. Making changes for FWD build.xml script to automatically create terminfo.zip file for FWD/build/lib directory. But only for Linux distribution.

Does the archive target for deploy also include the terminfo.zip?

This option has been added.

3. When some project calls ant deploy.prepare this zip file is copied into the same directory that has libp2j.so.

Has ant deployapp.prepare been tested? This does not perform a copy of the FWD libraries.

These steps above are already implemented and tested.

This one is still in progress.

Is there a branch where these are available?

Not yet. The task is under development testing different implementation approaches. When it is stable I'll create a branch to commit the changes.

One possible implementation approach is to use unzip tool inside native code to extract the terminfo archive. This would assume the unzip is installed on production system. Is it acceptable requirement/constraint?

Or we need to implement unzipping inside FWD?

Greg, Constantin,

What do you think?

To be safe, you can use jar, which allows you to extract from a zip file:

Usage: jar {ctxui}[vfmn0PMe] [jar-file] [manifest-file] [entry-point] [-C dir] files ...
Options:
    -c  create new archive
    -t  list table of contents for archive
    -x  extract named (or all) files from archive
    -u  update existing archive
    -v  generate verbose output on standard output
    -f  specify archive file name
    -m  include manifest information from specified manifest file
    -n  perform Pack200 normalization after creating a new archive
    -e  specify application entry point for stand-alone application 
        bundled into an executable jar file
    -0  store only; use no ZIP compression
    -P  preserve leading '/' (absolute path) and ".." (parent directory) components from file names
    -M  do not create a manifest file for the entries
    -i  generate index information for the specified jar files
    -C  change to the specified directory and include the following file
If any file is a directory then it is processed recursively.
The manifest file name, the archive file name and the entry point name are
specified in the same order as the 'm', 'f' and 'e' flags.

Example 1: to archive two class files into an archive called classes.jar: 
       jar cvf classes.jar Foo.class Bar.class 
Example 2: use an existing manifest file 'mymanifest' and archive all the
           files in the foo/ directory into 'classes.jar': 
       jar cvfm classes.jar mymanifest -C foo/ .

It is a good idea to use jar.

I'm also OK if we want to modify the client.sh and server.sh scripts instead of calling this from Java. In the end, since we are "building" a terminfo.zip along with the rest of our build, having the standard scripts check for presence and deploy if needed is a reasonable approach.

Constantin's points about the location are correct. We can provide an option to override the location in the script. This might be used for the case where a common location is used.

Finally, by default, this should not make any changes. It will be common for production systems to disallow file system changes exxcept from very specific paths (e.g. for logging).

Greg Shah wrote:

It is a good idea to use jar.

I'm also OK if we want to modify the client.sh and server.sh scripts instead of calling this from Java. In the end, since we are "building" a terminfo.zip along with the rest of our build, having the standard scripts check for presence and deploy if needed is a reasonable approach.

Constantin's points about the location are correct. We can provide an option to override the location in the script. This might be used for the case where a common location is used.

Finally, by default, this should not make any changes. It will be common for production systems to disallow file system changes exxcept from very specific paths (e.g. for logging).

The jar command can be used to extract zip file. But it has one disadvantage - you can not specify destination directory for extracting. So I had to add some directory manipulation code to handle this. But it worth.

So now I have the code that handling all we want in FWD. No changes for client.sh and/or server.sh are necessary. The only we need is to deploy somehow terminfo.zip into directory defined in -Djava.library.path.

Now I'm adding mutex lock for unpacking code and will commit initial changes for review.

Created task branch 5568a from trunk revision 15654.

Preparing to commit FWD related updates.

Guys,

I'm investigating the possible ways when terminfo.zip is not delivered for the production system. Can someone tell what ant commands are used in this case for project applicaton?

Is the command ant deploy.prepare not used? In this case how the FWD runtime is setting up on that system?

Eugenie Lyzenko wrote:

Guys,

I'm investigating the possible ways when terminfo.zip is not delivered for the production system. Can someone tell what ant commands are used in this case for project applicaton?

Is the command ant deploy.prepare not used? In this case how the FWD runtime is setting up on that system?

The customers can't be dependent on our sample build.xml scripts - these are just examples. Most (if not all) customers will have their own deploy scripts, docker build scripts, etc.

I will look at the code changes later, but I think the checking and unzipping of the terminfo should probably be in our bash scripts. I'm worried we will hard code too much deployment knowledge into the Java code.

Constantin Asofiei wrote:

Eugenie Lyzenko wrote:

Guys,

I'm investigating the possible ways when terminfo.zip is not delivered for the production system. Can someone tell what ant commands are used in this case for project applicaton?

Is the command ant deploy.prepare not used? In this case how the FWD runtime is setting up on that system?

The customers can't be dependent on our sample build.xml scripts - these are just examples. Most (if not all) customers will have their own deploy scripts, docker build scripts, etc.

So this is out of our control? But in any case the customer will use our dist/fwd_deploy*.zip archives or something like this, right?

Roger Borrello wrote:

Eugenie Lyzenko wrote:

So this is out of our control? But in any case the customer will use our dist/fwd_deploy*.zip archives or something like this, right?

I didn't finish my thought....

Doesn't java.library.path contain a list of directories? Like java.library.path = '/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib'?

Positioning the terminfo.zip so that it is available during runtime has to be a little more clever.

Roger Borrello wrote:

I received the below when I checked out 5568a and ran ant all sheet:war archive... I know it was PICNIC error (Problem In Chair Not In Computer), but wanted to document my failure:
[...]

So I went into src/native and ran SONAME=libp2j.so SRCDIR=$HOME/projects/fwd/5568a/src/native JVM_INCPATH=/usr/lib/jvm/java-8-openjdk-amd64/jre/.. make getting:
[...]

So obviously I need to follow Eugenie's instructions to #5568-31 to get NCURSES setup properly first.

My question is, do you have 2 NCURSES for your development of this task, or do you have to go back-and-forth with the #2660-12 steps?

I have more than 2 statically built NCURSES, legacy(to use with current trunk) and other for combinations wide char and pthread featured. All in different directories. To switch between versions I use soft link pointed to one I'm currently working with. Just edit the symlink and you do not even need to reset console session. It is convenient(at least for me).

The 5568a updated for review to revision 15656. Rebased with recent trunk 15658, new revision is 15660.

Small rework for using getcwd() for current directory value. The get_current_dir_name() is an extension and has no generic support for all distributions. Also little change for directory construction. The lib path is coming from Java side already ending with /.

But I'm also inclining to think we need to move archive extracting code to shell script starting the client. The ChUI regression testing does not properly work with this branch, the client is not starting. I'm looking for a reason.

The problem I see with handling TERMINFO_DIRS in bash script is the application server clients starting. As far as I know they are spawned ChUI sessions without client.sh usage. If we do everything in client.sh the TERMINFO for spawned client will be inconsistent I think.

Eugenie Lyzenko wrote:

The 5568a updated for review to revision 15656. Rebased with recent trunk 15658, new revision is 15660.

Small rework for using getcwd() for current directory value. The get_current_dir_name() is an extension and has no generic support for all distributions. Also little change for directory construction. The lib path is coming from Java side already ending with /.

But I'm also inclining to think we need to move archive extracting code to shell script starting the client. The ChUI regression testing does not properly work with this branch, the client is not starting. I'm looking for a reason.

The problem I see with handling TERMINFO_DIRS in bash script is the application server clients starting. As far as I know they are spawned ChUI sessions without client.sh usage. If we do everything in client.sh the TERMINFO for spawned client will be inconsistent I think.

On the other hand for spawned application server clients and batched clients there is no user input from keyboard so we do not need to patch TERMINFO in this case and client can use system wide defined terminal info data.

I think the current code puts a lot of "installation knowledge" into our native C code. That means that the installations will be very limited by that code instead of allowing flexibility for the devops or sysadmin people.

I think we should remove that and do the checking and setup of TERMINFO_DIRS as OPTIONAL step in the scripts. By default we should do nothing. Many customers will separately setup the terminfo dir using Docker or whatever devops tools they use (e.g. Ansible). We need to allow this to be placed anywhere and we definitely don't want every launch of a client to do an unzip (from the scripts or from our native code). That is costly and messy.

But I'm also inclining to think we need to move archive extracting code to shell script starting the client. The ChUI regression testing does not properly work with this branch, the client is not starting. I'm looking for a reason.

Yes, let's remove it. But I'm not sure we want to do the unzipping in the script.

The problem I see with handling TERMINFO_DIRS in bash script is the application server clients starting. As far as I know they are spawned ChUI sessions without client.sh usage. If we do everything in client.sh the TERMINFO for spawned client will be inconsistent I think.

Fair point. How should we set TERMINFO_DIRS in that case? I hesitate to put that into the native code.

I would prefer to minimize as much as possible the user interaction in TERMINFO_DIRS setup process to avoid human factor in this option config.

I don't think we can assume that all customers will allow TERMINFO_DIRS to be in that location. In other words, we need to allow customers the flexibility to put it anywhere they want OR EVEN to not use TERMINFO_DIRS (i.e. to rely upon the system terminfo).

The next step could be to put TERMINFO data directly into native code sources and use C code to create terminfo(/v|/x) dirs and files with respective content just based on libp2j.so code. This way we can avoid any dependencies(even from location of the terminfo.zip file).

This is interesting if it is possible.

We can't remove the patched ncurses support in this branch. Otherwise when customers move to this revision, they will be forced to make changes to their installations and they may not have the time to do that. So let's please let this be configurable (pthreads OR patching) at our build time. If using the threading approach, then we can drop the auto_getch_refresh and so forth. If using the patching approach, then auto_getch_refresh remains.

The 5568a updated for review to revision 15666.

Just clean up from commented code used in debug/testing.

Greg,

For the TERMINFO seems like I've found a way to integrate TERMINFO data into libp2j.so code. So I think we can be remove dependency from terminfo.zip. We will not even need to have this file in FWD. For your notes I think we can read boolean flag from directory.xml and pass it to native code to instruct library if we need to patch TERMINFO or not (use system wide TERMINFO). Do you agree with this approach?

For the TERMINFO seems like I've found a way to integrate TERMINFO data into libp2j.so code. So I think we can be remove dependency from terminfo.zip. We will not even need to have this file in FWD.

Interesting! Please describe how this will work.

For your notes I think we can read boolean flag from directory.xml and pass it to native code to instruct library if we need to patch TERMINFO or not (use system wide TERMINFO). Do you agree with this approach?

I'm not sure we need a flag. We could add a parameter to the cfgOverrides for this.

Greg Shah wrote:

For the TERMINFO seems like I've found a way to integrate TERMINFO data into libp2j.so code. So I think we can be remove dependency from terminfo.zip. We will not even need to have this file in FWD.

Interesting! Please describe how this will work.

There will be 3 const byte data arrays: xterm, vt220 and vt320. When it is time to initConsole we check if directory like base_dir/terminfo/v exists, if not we create it. Then we crate binary file like vt320 and write respective data into the file.

The required arrays already prepared. Working on logic to create directory/file.

For your notes I think we can read boolean flag from directory.xml and pass it to native code to instruct library if we need to patch TERMINFO or not (use system wide TERMINFO). Do you agree with this approach?

I'm not sure we need a flag. We could add a parameter to the cfgOverrides for this.

OK. I just wanted to note we need to pass the flag from Java to native code.

Wouldn't it be easier to do this from Java code? We could (as part of the build) include the terminfo/ directory structure (and files) in the p2j.jar and then just read those files from the jar and write then into the filesystem.

As noted before, by default we would do none of this.

(as part of the build) include the terminfo/ directory structure (and files) in the p2j.jar

My idea with this is to avoid having to hard code byte arrays inside our source. This should just create the content during the build and store it in the jar. It seems easier and less fragile since it can pick up OS level changes without us changing our code.

Greg Shah wrote:

Wouldn't it be easier to do this from Java code? We could (as part of the build) include the terminfo/ directory structure (and files) in the p2j.jar and then just read those files from the jar and write then into the filesystem.

Honestly I guess it would not be easier to handle this in Java code. I think the only case when data embedding in libp2j.so is not convenient is the case when terminfo system files are changing. I mean the original terminal definitions taken with command infocmp xterm > xterm.tmp" for xterm.tmp. Is it possible this content is changing from one Linux version to another?

On the other hand it is not a big deal to move terminfo data into *.c byte array, taken max 1 hour for all 3 files for me.

As noted before, by default we would do none of this.

So if none option is defined the FWD will not use TERMINFO_DIRS override with patched data, right?

The 5568a updated for review to revision 15667. Rebased with recent trunk 15663, new revision is 15668.

The changes have:
1. Notes resolution
- Adding condition compilation to be able to build FWD used patched NCURSES and no pthreads enabled static library. This is default building for given branch. To use advanced pthread build the environment variable export NCURSES_FWD_PTHREADS=yes should be defined in .bashrc.
- The using of TERMINFO_DIRS override is now optional. And turned off by default. To use the config for client script should be changed to have client:chui:terminfo_ovr=true defined.
2. Implemented experimental TERMINFO_DIRS override in native code. The target directory is the same as for libp2j.so. No need to have/make terminfo.zip file. This is for how it works demo. If there are objections we can roll it back. Meantime I'll do some ChUI regression testing.

I think the only case when data embedding in libp2j.so is not convenient is the case when terminfo system files are changing.

Which can happen at any time. We don't want to have code changes or even 1 minute of work to accomodate such changes.

I mean the original terminal definitions taken with command infocmp xterm > xterm.tmp" for xterm.tmp. Is it possible this content is changing from one Linux version to another?

Yes, they can be different.

On the other hand it is not a big deal to move terminfo data into *.c byte array, taken max 1 hour for all 3 files for me.

That is effort that is wasted.

As noted before, by default we would do none of this.

So if none option is defined the FWD will not use TERMINFO_DIRS override with patched data, right?

Yes, OR the TERMINFO_DIRS override is handled by the customer's scripting.

Greg Shah wrote:

I think the only case when data embedding in libp2j.so is not convenient is the case when terminfo system files are changing.

Which can happen at any time. We don't want to have code changes or even 1 minute of work to accomodate such changes.

I mean the original terminal definitions taken with command infocmp xterm > xterm.tmp" for xterm.tmp. Is it possible this content is changing from one Linux version to another?

Yes, they can be different.

On the other hand it is not a big deal to move terminfo data into *.c byte array, taken max 1 hour for all 3 files for me.

That is effort that is wasted.

In this case I would suggest to prepare the patched terminfo directory content on the FWD building level(build.xml):
- extract current system wide options by infocmp xterm|vt220|vt320
- make required patches for extracted tmp files
- pack the results into p2j.jar

Then if override option is specified Java code will read these data, make required directories on file system and set up new value for TERMINFO_DIRS.

Is this plan OK?

Is this plan OK?

Yes

The 5568a updated for review to revision 15670.

This update changes only build.xml. But all preparation steps to make patched TERMINFO binary files and pack them into p2j.jar is inside build.xml. So you can review the preparation approach while I do final steps for FWD runtime code.

The idea for preparation is to have backup copies of the patched TEMINFO binaries and use the copy files in the cases when the system wide TERMINFO file can not be retrieved. For example the vt320 data for the system there is now such terminal installed(this is true when NCURSES was never patched with dynamic version).

Also the xterm patch sed command was changed to reflect the TERMINFO current data. The reason is the system wide data does not have kbs=^H entry. Instead it now has kbs=^? and now it is what we need to replace with kbs=\177. The old sample was also kept for compatibility.

Continue working with FWD runtime changes.

The 5568a updated for review to revision 15671.

Fixed building issue for Windows with condition check.

The 5568a updated for review to revision 15672. Rebased with trunk revision 15671, new revision is 15679.

This is the FWD part of new TERMINFO_DIRS implementation approach.
1. Deleted src/natve/terminfo.c file. The hardcoded binary data for patched TERMINFO will not be used.
2. The Java client part is reworked. Now client uses p2j.jar to get patched TERMINFO data on demand(depending on client option which is off by default).
3. The Java client uses native code for 2 tasks:
- First to get the directory the native library is loading from. This become the base for creating patched TERMINFO tree in Java code.
- Second to override TERMINFO_DIRS environment variable. If there is a current value the new one is setting up in a head of the dir list.

Starting to run the ChUI regression testing.

Code Review Task Branch 5568a Revisions 15672 through 15679

Overall, it is good.

1. In build.xml, we should either move the temp terminfo directory into build/ or add a clean step to remove the temp terminfo dir.

2. It is OK to default to the libp2j.so directory BUT we should allow the user to optionally override the directory to use.

3. ConsoleDriver.makeTerminfoFile() will need more error handling for cases like when the filesystem is read-only (e.g. you can't mkdir() or otherwise edit it).

Greg Shah wrote:

Code Review Task Branch 5568a Revisions 15672 through 15679

Overall, it is good.

1. In build.xml, we should either move the temp terminfo directory into build/ or add a clean step to remove the temp terminfo dir.

OK. Changed to build/ directory. Will commit in next update.

2. It is OK to default to the libp2j.so directory BUT we should allow the user to optionally override the directory to use.

Agreed. How do you think it could be set? I would suggest the user to introduce optional system variable like FWD_TERMINFO_DIRS. Then FWD runtime will check if this var is defined. In cases when not defined we will use libp2j.so directory. Is this plan OK?

3. ConsoleDriver.makeTerminfoFile() will need more error handling for cases like when the filesystem is read-only (e.g. you can't mkdir() or otherwise edit it).

OK. Working on this.

2. It is OK to default to the libp2j.so directory BUT we should allow the user to optionally override the directory to use.

Agreed. How do you think it could be set? I would suggest the user to introduce optional system variable like FWD_TERMINFO_DIRS. Then FWD runtime will check if this var is defined. In cases when not defined we will use libp2j.so directory. Is this plan OK?

The user can pass it as a bootstrap cfg value.

Greg Shah wrote:

2. It is OK to default to the libp2j.so directory BUT we should allow the user to optionally override the directory to use.

Agreed. How do you think it could be set? I would suggest the user to introduce optional system variable like FWD_TERMINFO_DIRS. Then FWD runtime will check if this var is defined. In cases when not defined we will use libp2j.so directory. Is this plan OK?

The user can pass it as a bootstrap cfg value.

So this will be the user responsibility to provide valid directory, right?

Yes

The 5568a updated for review to revision 15680.

This is the notes 1 and 2 resolutions and some minor code improvements. Working on the error handling for ConsoleDriver.makeTerminfoFile().

The 5568a updated for review to revision 15681.

This is note 3 resolution with added error handling/logging for ConsoleDriver.makeTerminfoFile(). In a case of serious failure in TERMINFO_DIRS override setup the system wide TERMINFO will be used.

The 5568a rebased with trunk revision 15676. New revision is 15686.

Greg Shah wrote:

Code Review Task Branch 5568a Revisions 15685 and 15686

The changes look good.

I think we should expose code that implements the ConsoleDriver.initTerminfoDirs() (including ConsoleDriver.makeTerminfoFile()) as a separate utility. Let's move it out of ConsoleDriver and create a new Java class for it. Please add a main() so that it can easily be called from either ConsoleDriver OR from the command line. The user should be able to pass the directory name to the utility and it can do its thing. I think most customers will just integrate the command line program into their setup automation instead of letting FWD "auto-setup". This change will make that possible.

OK.

The 5568a rebased with trunk revision 15677. New revision is 15687. Just rebase.

Greg Shah wrote:

Code Review Task Branch 5568a Revisions 15685 and 15686

The changes look good.

I think we should expose code that implements the ConsoleDriver.initTerminfoDirs() (including ConsoleDriver.makeTerminfoFile()) as a separate utility. Let's move it out of ConsoleDriver and create a new Java class for it. Please add a main() so that it can easily be called from either ConsoleDriver OR from the command line. The user should be able to pass the directory name to the utility and it can do its thing. I think most customers will just integrate the command line program into their setup automation instead of letting FWD "auto-setup". This change will make that possible.

I'm going to create new class TerminfoHelper inside package com.goldencode.p2j.ui.client.chui.driver.console for this purpose.

Is it OK? Or we need to consider other name (and|or) class location?

I'm going to create new class TerminfoHelper inside package com.goldencode.p2j.ui.client.chui.driver.console for this purpose.

Is it OK? Or we need to consider other name (and|or) class location?

Yes, both are good.

Greg Shah wrote:

I'm going to create new class TerminfoHelper inside package com.goldencode.p2j.ui.client.chui.driver.console for this purpose.

Is it OK? Or we need to consider other name (and|or) class location?

Yes, both are good.

Let me clarify one more point.

In a case the user makes TERMINFO_DIRS setup as a standalone command line process(meaning the patched TERMINFO_DIRS tree structure is OK before main client is starting) what will be the next steps for client? When the override flag is set the client should just use already prepared TERMIFO_DIRS, not making new binary files from p2j.jar patched data? Right? We should assume the data prepared with this way are ready to be used in ChUI client sessions?

In a case the user makes TERMINFO_DIRS setup as a standalone command line process(meaning the patched TERMINFO_DIRS tree structure is OK before main client is starting) what will be the next steps for client? When the override flag is set the client should just use already prepared TERMIFO_DIRS, not making new binary files from p2j.jar patched data? Right?

Correct

We should assume the data prepared with this way are ready to be used in ChUI client sessions?

Yes, if it aleady exists we should assume it is OK and use it.

Greg Shah wrote:

We should assume the data prepared with this way are ready to be used in ChUI client sessions?

Yes, if it aleady exists we should assume it is OK and use it.

OK.

I have one more suggestion. For given terminfo file name we could create another empty file in same directory after binary setup completed. Say for xterm we create xterm.lck. This is additional protection. If the *.lck exists this means the TerminfoHelper setup completed (as part of the auto or command line setup) and we can bypass I/O operations for binary file. Also this way we can avoid files overwrite for further new ChUI sessions.

Do you think it is reasonable or not?

No, let's not make it more complicated.

If the directory exists and has any contents, then we should not try to add anything.

Greg Shah wrote:

No, let's not make it more complicated.

If the directory exists and has any contents, then we should not try to add anything.

OK. If this case if the TERMINFO file in target directory already exists we will assume it has valid content. Correct?

Eugenie Lyzenko wrote:

Greg Shah wrote:

No, let's not make it more complicated.

If the directory exists and has any contents, then we should not try to add anything.

OK. If this case if the TERMINFO file in target directory already exists we will assume it has valid content. Correct?

Yes

The 5568a updated for review to revision 15688.

This is the notes resolution for moving TERMINFO_DIRS handling in standalone class. It can be called with only option - new base directory to do TERMINFO patched tree.

Further if the terminfo file already exists and have the same size as one we have in p2j.jar the file will not be overwritten and considered as valid for terminal usage.

Will do the next regression testing round for version 15688.

The 5568a rebased with trunk revision 15678. New revision is 15689.

The 5568a rebased with trunk revision 15682. New revision is 15693.

Code Review Task Branch 5558a Revision 15693

1. The TerminfoHelper can be simplified.

• It does not need to be a singleton. The functionality of TerminfoHelper can be exposed as a single method public static boolean initTerminfoDirs(String baseDir). We don't need an instance of TerminfoHelper because there is no value in making multiple calls to it or storing any state. This is just a helper utility, called once.
• The use of ConsoleHelper is only needed when called from ConsoleDriver, not when used from main() so let's move the usage of ConsoleHelper back into ConsoleDriver. This ensures that we don't have to load the libp2j.so when running as a standalone utility.

2. Please rename ConsoleHelper.getNatveLibDir() to ConsoleHelper.getNativeLibDir().

3. Please rename ConsoleHelper.getDefaultConsoleHelper() to ConsoleHelper.getConsoleHelper(). There is no non-default ConsoleHelper so the name getDefaultConsoleHelper() is confusing and longer than needed.

Greg Shah wrote:

Code Review Task Branch 5558a Revision 15693

1. The TerminfoHelper can be simplified.

• It does not need to be a singleton. The functionality of TerminfoHelper can be exposed as a single method public static boolean initTerminfoDirs(String baseDir). We don't need an instance of TerminfoHelper because there is no value in making multiple calls to it or storing any state. This is just a helper utility, called once.
• The use of ConsoleHelper is only needed when called from ConsoleDriver, not when used from main() so let's move the usage of ConsoleHelper back into ConsoleDriver. This ensures that we don't have to load the libp2j.so when running as a standalone utility.

2. Please rename ConsoleHelper.getNatveLibDir() to ConsoleHelper.getNativeLibDir().

3. Please rename ConsoleHelper.getDefaultConsoleHelper() to ConsoleHelper.getConsoleHelper(). There is no non-default ConsoleHelper so the name getDefaultConsoleHelper() is confusing and longer than needed.

The 5568a updated for review to revision 15694. Rebased with trunk revision 15683, new revision is 15695.

This is the last notes resolution update.

The 5568a rebased with trunk revision 15684. New revision is 15696.

Please test ChUI regression testing, batch mode and (simple) appserver support.

Also: please document the additional OS installation requirements for building FWD on Linux.

Code Review Task Branch 5568a Revision 15695

I'm good with the changes.

So currently we use system wide TERMINFO for BatchDriver(). The question is do we need to use TERMINFO_DIRS override for batch driver?

No.

Greg Shah wrote:

Also: please document the additional OS installation requirements for building FWD on Linux.

Is it OK to extent the section https://proj.goldencode.com/projects/p2j/wiki/Patching_TERMINFO with new instructions?

Eugenie Lyzenko wrote:

Greg Shah wrote:

Also: please document the additional OS installation requirements for building FWD on Linux.

Is it OK to extent the section https://proj.goldencode.com/projects/p2j/wiki/Patching_TERMINFO with new instructions?

Yes, but please also update:

Building FWD From Source
Application Server Installation
Client Installation

Greg Shah wrote:

Eugenie Lyzenko wrote:

Greg Shah wrote:

Also: please document the additional OS installation requirements for building FWD on Linux.

Is it OK to extent the section https://proj.goldencode.com/projects/p2j/wiki/Patching_TERMINFO with new instructions?

Yes, but please also update:

Building FWD From Source
Application Server Installation
Client Installation

Seems like I have no access rights to these pages.

Try the links now.

Greg Shah wrote:

Try the links now.

Yes, the links in #5568-93 work.

The 5568a rebased with trunk revision 15685. New revision is 15697.

Constantin,

Seems like I need the assistance from you with simple appserver testing config. The idea is to test the 5568a branch changes with spawned client(appserver) configured in directory.xml. The requirement is not to use batch mode because the TERMINFO_DIRS override feature works only for ConsoleDriver instance in ChUI mode. So I would like to start some simple appserver with ConsoleDriver in use on client side. Is it possible? Can you advice something?

Eugenie Lyzenko wrote:

Constantin,

Seems like I need the assistance from you with simple appserver testing config. The idea is to test the 5568a branch changes with spawned client(appserver) configured in directory.xml. The requirement is not to use batch mode because the TERMINFO_DIRS override feature works only for ConsoleDriver instance in ChUI mode. So I would like to start some simple appserver with ConsoleDriver in use on client side. Is it possible? Can you advice something?

Appserver only runs in batch mode as a kind of character session.

Greg Shah wrote:

Eugenie Lyzenko wrote:

Constantin,

Seems like I need the assistance from you with simple appserver testing config. The idea is to test the 5568a branch changes with spawned client(appserver) configured in directory.xml. The requirement is not to use batch mode because the TERMINFO_DIRS override feature works only for ConsoleDriver instance in ChUI mode. So I would like to start some simple appserver with ConsoleDriver in use on client side. Is it possible? Can you advice something?

Appserver only runs in batch mode as a kind of character session.

So in this case the appserver will use TERMINFO data provided on system-wide level. The TERMINFO_DIRS feature from FWD will not be used. The user should patch system TERMINFO outside of the FWD project(before application is starting). Is it correct understanding?

That is a problem. The admin should be able to manually (or via a script) sets up the private TERMINFO using TerminfoHelper and then use their cfgOverrides to set the override for their appserver agents.

Just tested with big customer application and appserver agents. Seems like works OK for both default and standalone defined TERMINFO_DIRS directories.

So if no objections for code changes in 15698 we can pit it back to Internal Test status.

Code Review Task Branch 5568a Revision 15698

With this change, we can simplify the code a bit. There is no reason to have the change in OutputManager, nor the change in BatchDriver. Instead, why not just move the code from BatchDriver and OutputManager into a single place in ConsoleDriver? It is the same code, just duplciated right now. Let's get rid of the duplication.

Greg Shah wrote:

Code Review Task Branch 5568a Revision 15698

With this change, we can simplify the code a bit. There is no reason to have the change in OutputManager, nor the change in BatchDriver. Instead, why not just move the code from BatchDriver and OutputManager into a single place in ConsoleDriver? It is the same code, just duplciated right now. Let's get rid of the duplication.

The 5568a updated for review to revision 15699.

The new notes resolution.

The 5568a rebased with trunk revision 15686. New revision is 15700.

Continue with ChUI regression testing.

I'd like understand how best to position Docker to handle this. Do we want to support TERMINFO_DIRS when building a Docker image?

Right now, NCURSES is patched in the base Ubuntu image build:

# install libffi - https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source#Install-libffi
# install ncurses - https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source#Install-NCURSES-Development-Headers-Linux-Only
# install ant - https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source#Install-Apache-Ant
# install Pluggable Authentication Modules - https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source#Install-Pluggable-Authentication-Modules-aka-PAM-Linux-only
# Step 16/32
RUN apt-get update && \
    apt-get install -y --no-install-recommends $RUNTIME_TOOLS && \
    rm -rf /var/lib/apt/lists/*

# Setup ncurses patching and perform it
# Step 17/32
COPY setup_ncurses6x.sh /tmp
# Step 18/32
RUN cd /tmp && /tmp/setup_ncurses6x.sh -sap /opt/ncurses
ENV NCURSES_FWD_STATIC=/opt/ncurses
# Step 19/32
RUN rm -fr /tmp/ncurses* /tmp/setup_ncurses6x.sh

This is the basis for the FWD Docker images. It is a tradeoff, because while it is then "baked" into the base image, there is more simplicity in the overall design. If you were to have a base image with, and a base image without, every image downstream would have double the possibilities (with/without PostGreSQL, Java 8 or Java 17, etc).

If we are to support TERMINFO_DIRS, I assume we shouldn't be patching NCURSES in the base image, but this would force us down yet another "patched or not-patched, if patched, static or dynamic..." as well as having protection in downstream processes that prevent a not-patched and no TERMINFO_DIRS implementation, where we'd have to use Eugenie's new built-in.

I need some clear direction for Docker.

For Docker, we always would want to do the following:

1. Use PTHREADS and STATIC linking for NCURSES.
2. Manually create the private TERMINFO directory using TerminfoHelper during the creation of the image.
3. Use scripts (e.g. client.sh) that set the TERMINFO_DIRS environment variable to point to the private terminfo directory.
4. Ensure that any clients launched using directory settings properly set the override using the bootstrap cfg.

Roger Borrello wrote:

I'd like understand how best to position Docker to handle this. Do we want to support TERMINFO_DIRS when building a Docker image?

Right now, NCURSES is patched in the base Ubuntu image build:
[...]

This is the basis for the FWD Docker images. It is a tradeoff, because while it is then "baked" into the base image, there is more simplicity in the overall design. If you were to have a base image with, and a base image without, every image downstream would have double the possibilities (with/without PostGreSQL, Java 8 or Java 17, etc).

If we are to support TERMINFO_DIRS, I assume we shouldn't be patching NCURSES in the base image, but this would force us down yet another "patched or not-patched, if patched, static or dynamic..." as well as having protection in downstream processes that prevent a not-patched and no TERMINFO_DIRS implementation, where we'd have to use Eugenie's new built-in.

I need some clear direction for Docker.

Note the using patched or not NCURSES and use system-wide or TERMINFO_DIRS override are completely independent tasks. You can combine patched NCURSES with or without patched TERMINFO or use TERMINFO_DIRS override. To get rid of NCURESES and TERMINFO patching you have to use PTHREADS capable FWD build and use TERMINFO_DIRS override feature.

Eugenie Lyzenko wrote:

Note the using patched or not NCURSES and use system-wide or TERMINFO_DIRS override are completely independent tasks. You can combine patched NCURSES with or without patched TERMINFO or use TERMINFO_DIRS override. To get rid of NCURESES and TERMINFO patching you have to use PTHREADS capable FWD build and use TERMINFO_DIRS override feature.

Yes, I do know that. I was making the point that we could create many more Docker images with many variables, if we support all modes.

Greg Shah wrote:

For Docker, we always would want to do the following:

1. Use PTHREADS and STATIC linking for NCURSES.
2. Manually create the private TERMINFO directory using TerminfoHelper during the creation of the image.
3. Use scripts (e.g. client.sh) that set the TERMINFO_DIRS environment variable to point to the private terminfo directory.
4. Ensure that any clients launched using directory settings properly set the override using the bootstrap cfg.

Will PTHREADS and STATIC linking for NCURSES be provided by an update to setup_ncurses6x.sh? I consider the gcd_systems project as the keeper of the script in roles/patch_ncurses/files/setup_ncurses6x.sh, so if there's an update, lets update that version. I do understand that there my be external links, but we need to keep it in source control.

I see instructions in the updated wiki for creating the TERMINFO directory: java -cp path/to/p2j.jar com.goldencode.p2j.ui.client.chui.driver.console.TerminfoHelper base/directory/path/. If I wanted the directory to be /opt/terminfo in the Docker image, would that be java -cp $FWD_LIB/build/lib/p2j.jar com.goldencode.p2j.ui.client.chui.driver.console.TerminfoHelper /opt/terminfo? Then we would export TERMINFO_DIRS=/opt/terminfo globally in the Docker image?

Lastly, projects that utilize directory.xml.template would include cfgOverrides updates as mentioned (client:chui:terminfo_ovr=true and client:chui:terminfo_dirs_base=/opt/terminfo) where appropriate. Can the terminfo_ovr be changed to terminfo_override or terminfo_over?

Roger Borrello wrote:

I see instructions in the updated wiki for creating the TERMINFO directory: java -cp path/to/p2j.jar com.goldencode.p2j.ui.client.chui.driver.console.TerminfoHelper base/directory/path/. If I wanted the directory to be /opt/terminfo in the Docker image, would that be java -cp $FWD_LIB/build/lib/p2j.jar com.goldencode.p2j.ui.client.chui.driver.console.TerminfoHelper /opt/terminfo? Then we would export TERMINFO_DIRS=/opt/terminfo globally in the Docker image?

The option for TerminfoHelper is the base directory. Meaning the root for terminfo/v and terminfo/x subdirectories. In your cases the correct command will be: ava -cp $FWD_LIB/build/lib/p2j.jar com.goldencode.p2j.ui.client.chui.driver.console.TerminfoHelper /opt/. That's why I called it "base". Otherwise you will have /opt/terminfo/terminfo/... tree.

Lastly, projects that utilize directory.xml.template would include cfgOverrides updates as mentioned (client:chui:terminfo_ovr=true and client:chui:terminfo_dirs_base=/opt/terminfo) where appropriate. Can the terminfo_ovr be changed to terminfo_override or terminfo_over?

Again you need to specify base directory, /opt/ in your case.

For other constants naming I have no preferences, I tried to minimize the name length. But if they should be changed let's discuss an accept the names will be fine for everyone.

The 5568a rebased with trunk revision 15688. Updated for review to revision 15703.

The small change adds the usage of the ConsoleDriver(BootstrapConfig) to BatchDriver constructor. Otherwise the compilation fails.

Can the terminfo_ovr be changed to terminfo_override or terminfo_over?

Go ahead and change terminfo_ovr to terminfo_override. Once testing is done we can get this merged.

Greg Shah wrote:

Can the terminfo_ovr be changed to terminfo_override or terminfo_over?

Go ahead and change terminfo_ovr to terminfo_override. Once testing is done we can get this merged.

The 5568a updated for review to revision 15704.

Update changes the option name to terminfo_override. Restarting the tests.

The rebase with trunk 15688 stops appservers working with TERMINFO_DIRS feature. The BatchDriver changes in trunk introduce some incompatible changes. Investigating.

The 5568a updated for review to revision 15705.

Fixed the regression from recent rebase. Not all codes from init() method should be disabled. We need to keep optional TERMINFO setup. ChUI regression testing will be restarted soon.

Code Review Task Branch 5568a Revisions 15703 through 15705

The changes are good.

The 5568a rebased with trunk revision 15692. New revision is 15709.

I guess we are ready to merge this into trunk.

Is all documentation updated with the final details?

In the commit post, we need to summarize the changes, please post your suggested text here.

Greg Shah wrote:

Is all documentation updated with the final details?

I think the still missing point is to use PTHREADS capable NCURSES static build to get rid of auto refresh disable patch for NCURSES.

In the commit post, we need to summarize the changes, please post your suggested text here.

I would suggest the following: "Added optional usage of patched TERMINFO binaries and TERMINFO_DIRS environment, using PTHREADS capable static NCURSES to get rid of auto_getch_reftesh patch."

Greg,

I have updated the Wiki documentations. But it will be great if you will review, not sure if it is clear enough. Also in one place I left one stub just because the change is not yet in trunk.

Please list (here) the specific pages you have edited. I will review them.

In the commit post, we need to summarize the changes, please post your suggested text here.

I would suggest the following: "Added optional usage of patched TERMINFO binaries and TERMINFO_DIRS environment, using PTHREADS capable static NCURSES to get rid of auto_getch_reftesh patch."

It should be more verbose:

• TERMINFO can now be optionally read from a private FWD-specific TERMINFO directory instead of the systemwide TERMINFO. If not configured to use the private directory, by default FWD will continue to use the systemwide TERMINFO. Refs #5568.
• During the FWD build, a statically linked PTHREADS capable NCURSES which has no auto_getch_refresh patch can be used instead of a patched NCURSES. This must be confgured explicitly in the FWD build and it requires NCURSES to have been built with threading activated which is not the default. Refs #2660.

Please see X, Y and Z for more details.

(X, Y and Z) are the wiki pages with the documentation.

Greg Shah wrote:

Please list (here) the specific pages you have edited. I will review them.

For TERMINFO: https://proj.goldencode.com/projects/p2j/wiki/Patching_TERMINFO
For building from sources: https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source/edit?section=18
For client installation: https://proj.goldencode.com/projects/p2j/wiki/Client_Installation/edit?section=19
Additions to static linking instructions: https://proj.goldencode.com/projects/p2j/wiki/Patching_NCURSES_Using_Static_Linking

Greg Shah wrote:

In the commit post, we need to summarize the changes, please post your suggested text here.

I would suggest the following: "Added optional usage of patched TERMINFO binaries and TERMINFO_DIRS environment, using PTHREADS capable static NCURSES to get rid of auto_getch_reftesh patch."

It should be more verbose:

• TERMINFO can now be optionally read from a private FWD-specific TERMINFO directory instead of the systemwide TERMINFO. If not configured to use the private directory, by default FWD will continue to use the systemwide TERMINFO. Refs #5568.
• During the FWD build, a statically linked PTHREADS capable NCURSES which has no auto_getch_refresh patch can be used instead of a patched NCURSES. This must be confgured explicitly in the FWD build and it requires NCURSES to have been built with threading activated which is not the default. Refs #2660.

Please see X, Y and Z for more details.

(X, Y and Z) are the wiki pages with the documentation.

Can we put such long multi-line text in -m "" option of the bzr commit command?

Eugenie Lyzenko wrote:

Can we put such long multi-line text in -m "" option of the bzr commit command?

Make a text file with all your text, even with spaces, etc. Then add -F <textfile> or --file=<textfile> to bzr commit

Example:

bzr commit -F ../mytext.txt

I've made substantial changes to the Using a Private TERMINFO Directory section. Please review (and there is a TODO for you there).

I've made smaller edits in the other documents, but have left behind a TODO for you in Patch NCURSES Linux Only.

I did not edit Patching NCURSES Using Static Linking. I would like you to make more significant edits there. If you read that document, you would not really understand that using PTHREADS is:

1. The preferred way to configure the system.
2. Completely avoids the need to patch ncurses.

It is just not clear from the page right now. Actually, I think it needs to be a completely NEW page that provides a simple description for how to enable PTHREADS and properly build libp2j.so using that version. Then the references to NCURSES elsewhere need to highlight that page as the preferred way to setup the system.

The 5568a rebased with trunk revision 15696. New revision is 15713.

Greg Shah wrote:

I've made substantial changes to the Using a Private TERMINFO Directory section. Please review (and there is a TODO for you there).

I've made smaller edits in the other documents, but have left behind a TODO for you in Patch NCURSES Linux Only.

I did not edit Patching NCURSES Using Static Linking. I would like you to make more significant edits there. If you read that document, you would not really understand that using PTHREADS is:

1. The preferred way to configure the system.
2. Completely avoids the need to patch ncurses.

It is just not clear from the page right now. Actually, I think it needs to be a completely NEW page that provides a simple description for how to enable PTHREADS and properly build libp2j.so using that version. Then the references to NCURSES elsewhere need to highlight that page as the preferred way to setup the system.

I have prepared the draft for new page dedicated to PTREADS capable building. I think we need to confirm the page title to avoid unnecessary create/delete page if the title is not good. I would suggest Building Tread-safe NCURSES Using Static Linking. Is it OK?

Only information related to patch-free building will be in this page. We can leave old static building page as alternative way(but not preferred).

The 5568a rebased with trunk revision 15697. New revision is 15714.

I think we need to confirm the page title to avoid unnecessary create/delete page if the title is not good. I would suggest Building Tread-safe NCURSES Using Static Linking. Is it OK?

Please use Using Thread-Safe NCURSES with Static Linking as the title.

Only information related to patch-free building will be in this page. We can leave old static building page as alternative way(but not preferred).

OK

Eugenie Lyzenko wrote:

Greg Shah wrote:

I've made substantial changes to the Using a Private TERMINFO Directory section. Please review (and there is a TODO for you there).

I've made smaller edits in the other documents, but have left behind a TODO for you in Patch NCURSES Linux Only.

I have modified this section adding more details.

Including new section for PTHREADS aware static compilation which become the preferable way to use static link case. The new section draft has been also added for review. Similar to old static case but content is changed to have only PTHREADS references. Also I left one TODO note there. The question is do we need to have automated script to handle new build style? Or may be we need to modify existed setup_ncurses6x.sh to have additional option to use pthread (this is also can be made as default choice).

The 5568a rebased with trunk revision 15703. New revision is 15720.

The 5568a rebased with trunk revision 15709. New revision is 15726.

Regression testing in progress.

Meantime I have updated documents pages and added modified script as attachment. Please review Wiki.

The 5568a rebased with trunk revision 15711. New revision is 15728.

Roger: Please review and test the updated patch script.

The 5568a rebased with trunk revision 15713. New revision is 15730.

I've reviewed and revised the NCURSES documentation. See:

Client Installation (uses common text from NCURSES)
Building FWD From Source (uses common text from NCURSES)

That common text in NCURSES references these:

Patching NCURSES
Patching NCURSES Using Static Linking
Using Thread-Safe NCURSES with Static Linking

Eugenie: Please review. These pages will need editing once we have the revision number for the merge.

The Building FWD From Source needs to be edited to include the things needed for TERMINFO to be properly built with FWD.

Eugenie: Please make those edits.

Greg Shah wrote:

The Building FWD From Source needs to be edited to include the things needed for TERMINFO to be properly built with FWD.

Eugenie: Please make those edits.

Added section to describe this.

What is currently worry me is having several different versions of the automated script on different pages to download. I think we finally should have only one in Redmine to avoid confusion for which one is last/correct.

Eugenie Lyzenko wrote:

Greg Shah wrote:

The Building FWD From Source needs to be edited to include the things needed for TERMINFO to be properly built with FWD.

Eugenie: Please make those edits.

Added section to describe this.

What is currently worry me is having several different versions of the automated script on different pages to download. I think we finally should have only one in Redmine to avoid confusion for which one is last/correct.

• Check to prevent error message when /etc/profile.d/10-ncurses.sh doesn't exist
• Update to /etc/sudoers.d/10-ncurses so that the NCURSES_FWD_STATIC environment variable is available when patching under sudo
• mode control on the /etc/profile.d/10-ncurses.sh and $HOME'/.bashrc files
• Helpful comment in above files

Should it check to determine if NCURSES_FWD_PTHREADS=true or NCURSES_FWD_STATIC and exit gracefully, unless an overwrite option is passed in? That way we don't accidentally change the configuration.

Roger Borrello wrote:

Eugenie Lyzenko wrote:

Greg Shah wrote:

The Building FWD From Source needs to be edited to include the things needed for TERMINFO to be properly built with FWD.

Eugenie: Please make those edits.

Added section to describe this.

What is currently worry me is having several different versions of the automated script on different pages to download. I think we finally should have only one in Redmine to avoid confusion for which one is last/correct.

Shouldn't be in source control? There are a few significant items missing from the version that is in the gcd_systems project. That project is in ~/secure/code/gcd_systems in the ./roles/patch_ncurses/files/setup_ncurses6x.sh location.

The question. Why do we need to make .bashrc executable? Is the mode="+x" really necessary?

Eugenie Lyzenko wrote:

The question. Why do we need to make .bashrc executable? Is the mode="+x" really necessary?

Probably not. Actually, it should be 644, from my latest research. Do you concur?

Roger Borrello wrote:

Eugenie Lyzenko wrote:

The question. Why do we need to make .bashrc executable? Is the mode="+x" really necessary?

Probably not. Actually, it should be 644, from my latest research. Do you concur?

I have no objections. I just was confused why not executable file(.bashrc) should have executable attribute.

Eugenie Lyzenko wrote:

This is the updated script for review. Merged the changes from bzr repo.

The changes look good. I will test setting up a new system using it. However, what parameters should we use for standard workstation installs?

Roger Borrello wrote:

Eugenie Lyzenko wrote:

This is the updated script for review. Merged the changes from bzr repo.

The changes look good. I will test setting up a new system using it. However, what parameters should we use for standard workstation installs?

By default the thread-safe version is installing. If you need to have patched NCURSES version the -f option should be used with script.

Roger Borrello wrote:

Eugenie Lyzenko wrote:

This is the updated script for review. Merged the changes from bzr repo.

The changes look good. I will test setting up a new system using it. However, what parameters should we use for standard workstation installs?

The thread-safe version. It requires NO PATCHING, just a rebuild of NCURSES with PTHREADS enabled. That is a big deal.

Eugenie Lyzenko wrote:

Greg Shah wrote:

The Building FWD From Source needs to be edited to include the things needed for TERMINFO to be properly built with FWD.

Eugenie: Please make those edits.

Added section to describe this.

That section is not OK for the "build FWD" document. We need documentation about what we must do to enable the TERMINFO definitions being patch/compiled and included into the p2j.jar. We don't need the details of the runtime system changes and we don't need to understand how to use the private terminfo. But our FWD build now requires the terminfo tools to be there, right? And maybe other things? Those are new requirements for our build system. Please rework the doc there for this purpose.

Greg Shah wrote:

Eugenie Lyzenko wrote:

Greg Shah wrote:

The Building FWD From Source needs to be edited to include the things needed for TERMINFO to be properly built with FWD.

Eugenie: Please make those edits.

Added section to describe this.

That section is not OK for the "build FWD" document. We need documentation about what we must do to enable the TERMINFO definitions being patch/compiled and included into the p2j.jar. We don't need the details of the runtime system changes and we don't need to understand how to use the private terminfo. But our FWD build now requires the terminfo tools to be there, right? And maybe other things?

The only requirement is to have both infocmp and tic on development system. The rest is handling by FWD build. I thought it is a part of system-wide NCURSES dev package setup. So the system should have both out of the box. Am I wrong?

Those are new requirements for our build system. Please rework the doc there for this purpose.

Is this a new requirement? Should I note the requirement to ensure infocmp and tic are ready to use?

The only requirement is to have both infocmp and tic on development system. The rest is handling by FWD build. I thought it is a part of system-wide NCURSES dev package setup. So the system should have both out of the box. Am I wrong?

Yes, I think so. ;)

1. Our preference is that the customer would use the PTHREADS version of NCURSES AND that the system-wide NCURSES would NOT be changed. When we build from source code, I would expect us to build a local PTHREADS copy of NCURSES and use that for linking. If that is not properly documented, then we need to make that clear.
2. The TERMINFO tools have nothing to do with NCURSES itself and should be defined and supported separately. Yes, NCURSES uses the TERMINFO database but the tools are separate. Our installation should be separate. Most importantly, we have NEVER required the TERMINFO tools on our BUILD systems. They have only been required on the FWD client system at RUNTIME.

Those are new requirements for our build system. Please rework the doc there for this purpose.

Is this a new requirement? Should I note the requirement to ensure infocmp and tic are ready to use?

No, we don't want to say it that way.

Yes, it is now a new requirement for the BUILD system. So we need to add to the documentation but in a way that explains how to install those tools (apt-get install ...).

Greg Shah wrote:

The only requirement is to have both infocmp and tic on development system. The rest is handling by FWD build. I thought it is a part of system-wide NCURSES dev package setup. So the system should have both out of the box. Am I wrong?

Yes, I think so. ;)

1. Our preference is that the customer would use the PTHREADS version of NCURSES AND that the system-wide NCURSES would NOT be changed. When we build from source code, I would expect us to build a local PTHREADS copy of NCURSES and use that for linking. If that is not properly documented, then we need to make that clear.
2. The TERMINFO tools have nothing to do with NCURSES itself and should be defined and supported separately. Yes, NCURSES uses the TERMINFO database but the tools are separate. Our installation should be separate. Most importantly, we have NEVER required the TERMINFO tools on our BUILD systems. They have only been required on the FWD client system at RUNTIME.

Those are new requirements for our build system. Please rework the doc there for this purpose.

Is this a new requirement? Should I note the requirement to ensure infocmp and tic are ready to use?

No, we don't want to say it that way.

Yes, it is now a new requirement for the BUILD system. So we need to add to the documentation but in a way that explains how to install those tools (apt-get install ...).

Greg,

I have modified the document.

1. Added new section for install NCURES bin and data.
2. Made strictly note we should use thread-safe static library approach as primary and desirable.

Also I think the section https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source/edit?section=14 should be removed because we do not need header files to be on system-wide basis to build private NCURSES.

Please review again.

Also I think the section https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source/edit?section=14 should be removed because we do not need header files to be on system-wide basis to build private NCURSES.

As far as I know, we can't build the native library without the headers. Am I wrong?

Added new section for install NCURES bin and data.

Does this ensure that the TERMINFO tools infocmp and tic are installed?

Greg Shah wrote:

Added new section for install NCURES bin and data.

Does this ensure that the TERMINFO tools infocmp and tic are installed?

Yes.

Greg Shah wrote:

Also I think the section https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source/edit?section=14 should be removed because we do not need header files to be on system-wide basis to build private NCURSES.

As far as I know, we can't build the native library without the headers. Am I wrong?

Yes it is true. But we already have headers inside private NCURSES and we currently use the private headers at a top of the include dir, no need to additionally install system-wide.

Eugenie Lyzenko wrote:

Greg Shah wrote:

Also I think the section https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source/edit?section=14 should be removed because we do not need header files to be on system-wide basis to build private NCURSES.

As far as I know, we can't build the native library without the headers. Am I wrong?

Yes it is true. But we already have headers inside private NCURSES and we currently use the private headers at a top of the include dir, no need to additionally install system-wide.

We have to continue to support all 3 forms of NCURSES including the system-wide patched version. I think we can't assume the private NCURSES build.

Greg Shah wrote:

Eugenie Lyzenko wrote:

Greg Shah wrote:

Also I think the section https://proj.goldencode.com/projects/p2j/wiki/Building_FWD_From_Source/edit?section=14 should be removed because we do not need header files to be on system-wide basis to build private NCURSES.

As far as I know, we can't build the native library without the headers. Am I wrong?

Yes it is true. But we already have headers inside private NCURSES and we currently use the private headers at a top of the include dir, no need to additionally install system-wide.

We have to continue to support all 3 forms of NCURSES including the system-wide patched version. I think we can't assume the private NCURSES build.

OK. I replaced TODO with respective comments, we will keep it.

I've removed the TERMINFO section from the build FWD page. I think it is good to go.

Please edit the proposed text as described in #5568-125 and post the new proposed text here. Once agreed, we can move ahead with the merge.

Is there any reason we should not commit the patching script into FWD (tools/scripts/setup_ncurses6x.sh)?

Greg Shah wrote:

Is there any reason we should not commit the patching script into FWD (tools/scripts/setup_ncurses6x.sh)?

I do not see such reasons.

Greg Shah wrote:

I've removed the TERMINFO section from the build FWD page. I think it is good to go.

Please edit the proposed text as described in #5568-125 and post the new proposed text here. Once agreed, we can move ahead with the merge.

My propose with small changes to original:

- TERMINFO can now be optionally read from a private FWD-specific TERMINFO_DIRS directory instead of the systemwide TERMINFO. If not configured to use the private directory, by default FWD will continue to use the systemwide TERMINFO. Refs #5568.
- During the FWD build, a statically linked PTHREADS capable NCURSES which has no auto_getch_refresh patch can be used instead of a patched NCURSES. This must be confgured explicitly in the FWD build based on system NCURSES_FWD_PTHREADS variable definition and it requires NCURSES to have been built with threading activated which is not the default. Refs #2660.

Please see 'Building FWD From Source', 'Application Server Installation' and 'Client Installation' pages for more details.

Please see 'Building FWD From Source', 'Application Server Installation' and 'Client Installation' pages for more details.

In bzr commit, this text is not needed.

Do add these in the post in Redmine:

Please see:

Client Installation
Building FWD From Source

The common text is in NCURSES which references these:

Patching NCURSES
Patching NCURSES Using Static Linking
Using Thread-Safe NCURSES with Static Linking

In other words, in Redmine make sure the links are there. In bzr, we don't need those.

Greg Shah wrote:

In other words, in Redmine make sure the links are there. In bzr, we don't need those.

OK. The final text will be:

- TERMINFO can now be optionally read from a private FWD-specific TERMINFO_DIRS directory instead of the systemwide TERMINFO. If not configured to use the private directory, by default FWD will continue to use the systemwide TERMINFO. Refs #5568.
- During the FWD build, a statically linked PTHREADS capable NCURSES which has no auto_getch_refresh patch can be used instead of a patched NCURSES. This must be confgured explicitly in the FWD build based on system NCURSES_FWD_PTHREADS variable definition and it requires NCURSES to have been built with threading activated which is not the default. Refs #2660.

Greg Shah wrote:

You can merge after 6433a.

Don't forget to add tools/scripts/setup_ncurses6x.sh before you merge.

OK.

The 5568a rebased with trunk revision 15726. Updated to revision 15744 with adding script to tools/scripts/.

FYI: #6433 was just merged to trunk.

Alexandru Lungu wrote:

FYI: #6433 was just merged to trunk.

I see, thanks.

The 5568a rebased with trunk revision 15727. New revision is 15745.

Starting merge 5568a into trunk.

Branch 5568a was merged to trunk as revno 15728 then it was archived.

Currently the Docker image base ubuntu build has this step:

# Setup ncurses patching and perform it
# Step 17/32
COPY setup_ncurses6x.sh /tmp
# Step 18/32
RUN cd /tmp && /tmp/setup_ncurses6x.sh -sap /opt/ncurses
ENV NCURSES_FWD_STATIC=/opt/ncurses
# Step 19/32
RUN rm -fr /tmp/ncurses* /tmp/setup_ncurses6x.sh

Because the latest version of setup_ncurses6x.sh is contained within the FWD project, it has become a pre-requisite for other OS-setup processes, such as the container and gcd_systems. They also contain the setup_ncurses6x.sh script. Any suggestions on how to best position other projects, so they can retrieve a subset of the FWD project, so we don't need to have the other projects check in their own version?

The container project only exists for FWD, right? I'm not sure why it is even a separate thing.

As for gcd_systems, it is something private to GCD and if we need to check in a copy of the script that is fine.

Greg Shah wrote:

The container project only exists for FWD, right? I'm not sure why it is even a separate thing.

Wasn't there a customer that wanted it? Yes, it could be part of FWD, but the requirement to move it (which really would be just a few files in a docker directory) hadn't been given. I can scope it out.

As for gcd_systems, it is something private to GCD and if we need to check in a copy of the script that is fine.

OK.

There are some changes required to setup_ncurses6x.sh to work in the Docker build. For one thing, the previous version always created a statically patched ncurses. There isn't anything that would remove -fPIC from CFLAGS. It is always there. Should that have been an option?

Then there's the environment. The if [ -z "$NCURSES_FWD_STATIC" ]; then was there to prevent a second setup of the environment, not to make a determination as to whether or not NCURSES was statically patched. So the environment setup area is somewhat wrong. We really need 2 conditions related to NCURSES_FWD_STATIC and NCURSES_FWD_PTHREADS separately, because we can have both in the environment.

I am formulating an update.

I believe this update for the end of the setup_ncurses6x.sh would work:

# Set up environment variable(s) if they are not set already
env_update=""; sudo_update="" 
if [ -z "$NCURSES_FWD_STATIC" ]; then
   env_update+="export NCURSES_FWD_STATIC=\"$PWD\""$'\n'
   sudo_update+="Defaults env_keep += NCURSES_FWD_STATIC"$'\n'
fi
if [ -z "$NCURSES_FWD_PTHREADS" ] && [ "$patch_ncurses" != "true" ];
   env_update+="export NCURSES_FWD_PTHREADS=true"$'\n'
   sudo_update+="Defaults env_keep += NCURSES_FWD_PTHREADS"$'\n'
fi
if [ ."$env_update". != ".." ]; then
   if [[ "$system" == "true" ]]; then
      rc_filename='/etc/profile.d/10-ncurses.sh'
      msg="|         The environment has been changed! Reboot to activate changes         |" 
      # Ensure we are making the only entry in the profile.d script
      [ -f "$rc_filename" ] && $dry rm -f $rc_filename 
      [ -z "$dry" ] && echo -n "$sudo_update" > /etc/sudoers.d/10-ncurses
      mode="644" 
   else
      rc_filename=$HOME'/.bashrc'
      msg="|   The .bashrc has been changed! Please open new console to activate setup!   |" 
   fi
   if [[ -z "$dry" ]]; then
      echo '' >> $rc_filename
      if [[ "$patch_ncurses" == "true" ]]; then
         echo '# set up patched NCURSES root location' >> $rc_filename
      else
         echo '# set up thread-safe NCURSES root location' >> $rc_filename
      fi
      echo -n "$env_update" >> $rc_filename 
      if [[ "$system" == "true" ]]; then
         chmod $mode $rc_filename
      fi
   fi
   echo "|   ---------------------------   WARNING ! --------------------------------   |" 
   echo "$msg" 
   echo "|   ---------------------------   WARNING ! --------------------------------   |" 
fi

echo "|              Please do FULL FWD rebuild to activate the changes.             |" 
echo "+------------------------------------------------------------------------------+" 

# back to the starting directory
$dry popd > /dev/null

This doesn't address the fact that the script itself doesn't build a dynamically linked ncurses. Should that be added? I could create 5568b for that.

Roger Borrello wrote:

There are some changes required to setup_ncurses6x.sh to work in the Docker build. For one thing, the previous version always created a statically patched ncurses. There isn't anything that would remove -fPIC from CFLAGS. It is always there. Should that have been an option?

Then there's the environment. The if [ -z "$NCURSES_FWD_STATIC" ]; then was there to prevent a second setup of the environment, not to make a determination as to whether or not NCURSES was statically patched. So the environment setup area is somewhat wrong. We really need 2 conditions related to NCURSES_FWD_STATIC and NCURSES_FWD_PTHREADS separately, because we can have both in the environment.

It is not possible to have NCURSES_FWD_PTHREADS defined without NCURSES_FWD_STATIC. The thread-safe version of the private NCURSES must be statically linked with.

Eugenie Lyzenko wrote:

It is not possible to have NCURSES_FWD_PTHREADS defined without NCURSES_FWD_STATIC. The thread-safe version of the private NCURSES must be statically linked with.

That's OK. The code above makes that assumption, as well. It was more of the previous version that was only building static (before adding thread-safe option). There was only one build and that was static.

In any case, the above change (I modified the post) is better for Docker build, because the sudo environment needs to maintain the environment variables. Also, it's a little easier to follow the logic.

I created branch 5568b and checked in the updated setup_ncurses6x.sh.

Greg Shah wrote:

The container project only exists for FWD, right? I'm not sure why it is even a separate thing.

As for gcd_systems, it is something private to GCD and if we need to check in a copy of the script that is fine.

Regarding this question, while working the container project files into the docker directory of FWD, I realized the main benefit to having container as a separate project is the fact that I designed it so you can build many, many versions of FWD to put into their respective repositories with appropriate tags (<branch>_<revision>) very easily using available options to build_fwd.sh. In other words, I can have a single checkout of container and crank out builds from it by specifying the branch/revision to build. The scripting of builds can be centrally managed.

With the "project" within the FWD project, you have to checkout FWD, then build. Each checkout would drive the build of the Docker image. This wasn't in the design, and if you did want to centrally manage builds, you'd end up creating something like container, which would manage checkouts and drive Docker image builds. In the end, it's 6 of 1, 1/2 dozen of the other. However, one of those methods has already been designed and works, while the other would involve additional work, and a new project to boot.

Eugenie Lyzenko wrote:

I made some bug fixing and addition. Tested on Ubuntu 22.04. Please review new candidate.

I see you are trying to make the comment of the profile file accurate. That's good. I don't have a problem with this change.

I guess we can make direct commit to the trunk. No need to make another 5568b branch.

That's fine. I was working on the "docker" directory, which would have required another branch. I have not checked anything in besides the setup_ncurses6x.sh change to this point.

If Greg deems it necessary to merge the container project into FWD, I can create a new branch.

OK. Sorry, I missed the fact you already created new branch. So I'm late.

The 5568b upadted to revision 15734 with refined script.

Eugenie Lyzenko wrote:

OK. Sorry, I missed the fact you already created new branch. So I'm late.

The 5568b upadted to revision 15734 with refined script.

I built base and FWD images with the updated setup_ncurses6x.sh. I also converted and ran Hotel ChUI in Docker, so I think this is good.

Merge notification for task branch 5568b.

• Updates for handling Docker builds, as well as fixing a missing options in sudoers.
• Refined to consider partial updates.

This is only runtime change.

I notice that ant-native fails to build unless you have "NCURSES_FWD_PTHREADS=true". Is that by design that we only support that in FWD?

Roger Borrello wrote:

I notice that ant-native fails to build unless you have "NCURSES_FWD_PTHREAD=true". Is that by design that we only support that in FWD?

I guess in your case it is feature, not bug.

The native module linked with proper libraries. For thread-safe version the libraries should be suffixed with t like libncursest.a for patched FWD build the libraries must be without suffix (libncurses.a). Otherwise the FWD building will fail. And this is important because thread-safe version has internal mutex logic that is disabled in regular version.

If you want thread-safe version, NCURSES_FWD_PTHREAD=true should be specified and libraries should have t suffix.

Eugenie Lyzenko wrote:

Roger Borrello wrote:

I notice that ant-native fails to build unless you have "NCURSES_FWD_PTHREAD=true". Is that by design that we only support that in FWD?

I guess in your case it is feature, not bug.

The native module linked with proper libraries. For thread-safe version the libraries should be suffixed with t like libncursest.a for patched FWD build the libraries must be without suffix (libncurses.a). Otherwise the FWD building will fail. And this is important because thread-safe version has internal mutex logic that is disabled in regular version.

If you want thread-safe version, NCURSES_FWD_PTHREAD=true should be specified and libraries should have t suffix.

Got it... In trying to get the Docker images built, the environment was still not setup properly (I have to have the ENV NCURSES_FWD_PTHREADS=true in the Docker file to build FWD properly, and it was not set (because the Docker build environment doesn't have all the same environment as "normal"). I had made the update, but didn't have the 'S' on PTHREADS. :-( Sorry for that.

Eugenie, did you have a chance to test on Ubuntu 24.04? I see errors when it comes to handling the /etc/apt/sources.list using the -a option:

fwd@fwddev2:~/projects/mgmt$ cat /etc/apt/sources.list
# Ubuntu sources have moved to /etc/apt/sources.list.d/ubuntu.sources
fwd@fwddev2:~/projects/mgmt$ ls /etc/apt/sources.list.d/
ubuntu.sources  ubuntu.sources.curtin.orig
fwd@fwddev2:~/projects/mgmt$ cat /etc/apt/sources.list.d/ubuntu.sources
Types: deb
URIs: http://us.archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://security.ubuntu.com/ubuntu/
Suites: noble-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

So there is going to have to be an update to handle newer Ubuntu.

Roger Borrello wrote:

Eugenie, did you have a chance to test on Ubuntu 24.04? I see errors when it comes to handling the /etc/apt/sources.list using the -a option:

[...]

So there is going to have to be an update to handle newer Ubuntu.

This is a diff that I was able to use to setup NCURSES on 24.04 server:

45,46c45,51
<    $dry cp /etc/apt/sources.list /tmp
<    $dry sed -i '/^# deb-src /s/^# //' /etc/apt/sources.list
---
>    if [[ -f "/etc/apt/sources.list.d/ubuntu.sources" ]]; then
>       $dry cp /etc/apt/sources.list.d/ubuntu.sources /tmp
>       $dry sed -i '/^Types: deb$/a Types: deb-src' /etc/apt/sources.list.d/ubuntu.sources
>    else
>       $dry cp /etc/apt/sources.list /tmp
>       $dry sed -i '/^# deb-src /s/^# //' /etc/apt/sources.list
>    fi
58c63,67
<    $dry mv /tmp/sources.list /etc/apt/
---
>    if [[ -f "/etc/apt/sources.list.d/ubuntu.sources" ]]; then
>       $dry mv /tmp/ubuntu.sources /etc/apt/sources.list.d/
>    else
>       $dry mv /tmp/sources.list /etc/apt/
>    fi

It is mostly having to do with which file contains the sources definitions, and how to update them.

I expect this script to function across a range of recent Ubuntu releases. Let's get this finished and only have to check in one more time. This should have been caught earlier in this process.

We will not consider non LTS reseases?

Correct. We should support 18.04, 20.04, 22.04 and 24.04.

Eugenie Lyzenko wrote:

Greg Shah wrote:

I expect this script to function across a range of recent Ubuntu releases. Let's get this finished and only have to check in one more time. This should have been caught earlier in this process.

This is new candidate.

Except there are 4 spaces of indentation in your new section, instead of 3.

I have tested with 18.04 and 22.04.

Roger, can we consider the 24.04 is also verified? Do you have any install for 20.04?

I can do that. The mgmt project was updated with the new version, so I can just try to apply it to 20.04 install.

Since we aren't looking at specific versions, but rather how the sources listings are handled (/etc/apt/sources.list.d/ubuntu.sources or /etc/apt/sources.list), we may be better positioned for a broad spectrum of releases.

Roger Borrello wrote:

Eugenie Lyzenko wrote:

Greg Shah wrote:

I expect this script to function across a range of recent Ubuntu releases. Let's get this finished and only have to check in one more time. This should have been caught earlier in this process.

This is new candidate.

Except there are 4 spaces of indentation in your new section, instead of 3.

Fixed in new attachment.

I have tested with 18.04 and 22.04.

Roger, can we consider the 24.04 is also verified? Do you have any install for 20.04?

I can do that. The mgmt project was updated with the new version, so I can just try to apply it to 20.04 install.

Thank you. I'll get this possibility not early than tomorrow I think.

If everything will be OK we can do direct commit to trunk. No need to make 5568c branch.

Since we aren't looking at specific versions, but rather how the sources listings are handled (/etc/apt/sources.list.d/ubuntu.sources or /etc/apt/sources.list), we may be better positioned for a broad spectrum of releases.

I wouldn't be over-optimistic. At least we have one year+ till 26.04 release.

Please don't delete old versions of the script. Just add new ones. Redmine handles that just fine. If you delete the old one, then we lose all history of what was being discussed and only see the code for the final result. That isn't good.

Greg Shah wrote:

Please don't delete old versions of the script. Just add new ones. Redmine handles that just fine. If you delete the old one, then we lose all history of what was being discussed and only see the code for the final result. That isn't good.

OK. Sorry for inconvenience.

Eugenie Lyzenko wrote:

Thank you. I'll get this possibility not early than tomorrow I think.

I had success installing on 20.04. Ship it!

Roger Borrello wrote:

Eugenie Lyzenko wrote:

Thank you. I'll get this possibility not early than tomorrow I think.

I had success installing on 20.04. Ship it!

It is great news. Did you make first setup on that system? I mean no static @NCURSES was defined before install.

I can explain. I also experimented with truncated 20.04 - Lubuntu. And have an issue for default run(no options specified). The NCURSES root directory is setting up wrong, the ncurses-6.2 is not appending to environment variable. Instead it refers to directory the setup_ncurses6x.sh is running from.

Don't you see this issue for your 20.04?

Eugenie Lyzenko wrote:

Roger Borrello wrote:

Eugenie Lyzenko wrote:

Thank you. I'll get this possibility not early than tomorrow I think.

I had success installing on 20.04. Ship it!

It is great news. Did you make first setup on that system? I mean no static @NCURSES was defined before install.

I can explain. I also experimented with truncated 20.04 - Lubuntu. And have an issue for default run(no options specified). The NCURSES root directory is setting up wrong, the ncurses-6.2 is not appending to environment variable. Instead it refers to directory the setup_ncurses6x.sh is running from.

Don't you see this issue for your 20.04?

I did a fresh install of 20.04 on a VM, then installed mgmt which has an updated ncurses task, which uses our new version. I did have some Ansible troubles related to checking return codes, but that is not related.

Eugenie Lyzenko wrote:

I would suggest one more small change for setup script:

No... there may be spaces in the path. It could be enhanced to determine if the NCURSES_FWD_STATIC or NCURSES_FWD_PTHREADS are already in the file, and just update them (to make sure we don't have multiple instances).

Roger Borrello wrote:

Eugenie Lyzenko wrote:

I would suggest one more small change for setup script:

No... there may be spaces in the path. It could be enhanced to determine if the NCURSES_FWD_STATIC or NCURSES_FWD_PTHREADS are already in the file, and just update them (to make sure we don't have multiple instances).

OK. Let's keep the previous version. I see you point about spaces in a path.