Patching NCURSES¶
- Patching NCURSES
Introduction¶
FWD supports a Character User Interface (ChUI) on a "native" terminal interface, which will be used by 2 types of remote user:
- hardware terminals over serial ASCII links (a.k.a. TTY clients, dumb ASCII terminals)
- remote access via terminal emulator software (e.g. Putty)
Both of these native terminal implement some form of terminal type such as VT100 or VT220. The FWD support of these terminal clients requires NCURSES. A customized version of NCURSES 5.5 or later is required. NCURSES 5.5 is the minimum starting point since a bug has been found in the NCURSES getch() that affects process launching (getch improperly blanks the screen). This bug is fixed in NCURSES 5.5.
The reason for the NCURSES modification is to provide enough thread safety to enable a dual-threaded approach to using NCURSES. By default, NCURSES is not thread safe. The basic idea is that one thread may be dedicated to reading the keyboard via getch() while another thread is used for all output tasks. The NCURSES getch() function is used to read a keystroke. This is (at its core) an input function. Without this patch, calls to getch() will also attempt to flush any pending output to the terminal (which is inherently an output function). The problem is that the FWD ChUI client must independently process user input and output. In particular, the user input must be completely asynchronous to enable the CTRL-C processing in the ABL ChUI (CTRL-C can generate a STOP condition asynchronous to everything else). The state of the screen should not be affected by the keystroke reading code. This patch enables a mode where the caller can decouple the getch() keystroke reading from the output flushing (the output flushing becomes bypassed). It turns out that such an approach makes NCURSES very safe for usage by 2 threads. As long as one thread only ever handles the key reading (calls to getch()), the other thread can handle all output tasks and flushing.
All common Linux distributions have support for and include NCURSES binaries by default. NCURSES is also available in source form for many (if not all) UNIX platforms.
These are the instructions to patch, rebuild and install the necessary customized version of NCURSES.
Since trunk revision 14616 the NCURSES object libraries can optionally be statically linked to produce the libp2j.so. Using static linking means that a local copy of NCURSES is patched instead of the system-wide library. When using static linking in the FWD build the patched version of NCURSES is copied into libp2j.so, resulting in a larger library that contains all required patched NCURSES calls internally. Please see Patching NCURSES Using Static Linking for the details.
Obtain the Patches¶
Here are the download links for the NCURSES patches.
For NCURSES v5.5 and v5.6 on Linux, use:
ncurses_curses_h_in_20060828.patch
ncurses_lib_getch_c_20060828.patch
For versions of NCURSES from v5.7 to v6.0 on Linux, use:
ncurses_curses_h_in_20060828.patch
ncurses_lib_getch_c_v5.7_20090512.patch
For v5.9 in Solaris, use:
ncurses_curses_h_in_20060828.patch
ncurses_curses_h_in_v5.9_20160219.solaris.patch
For version of NCURSES from v6.1 on Linux, use:
ncurses_curses_h_in_v6.1_20200708.patch
ncurses_lib_getch_c_v6.1_20200708.patch
Prerequisites¶
Before patching NCURSES (manually or using scripts), you need to ensure that the source repositories are available for your Linux installation.
Source Repos¶
It is important that apt has access to the source code repositories.
Run cat /etc/apt/sources.list | grep deb-src to see the status of the source repositories. Check if all of the entries are commented out (with a # character at the beginning of the line. If you see something like this:
# deb-src http://us.archive.ubuntu.com/ubuntu/ xenial main restricted # deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates main restricted # deb-src http://us.archive.ubuntu.com/ubuntu/ xenial universe # deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates universe # deb-src http://us.archive.ubuntu.com/ubuntu/ xenial multiverse # deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates multiverse # deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted # deb-src http://security.ubuntu.com/ubuntu xenial-security universe # deb-src http://security.ubuntu.com/ubuntu xenial-security multiverse
Then you should remove the comment character, leaving behind the rest of the line.
Before changing the file, you can back it up using this command:
sudo cp /etc/apt/sources.list /etc/apt/sources.list.orig
And here is a command to make all the replacements for you:
sudo sed --in-place 's/# deb-src/deb-src/' /etc/apt/sources.list
Make sure dpkg-dev is installed. This package provides the development tools (including dpkg-source) required to unpack and build source packages. If not already installed, use
sudo apt-get install dpkg-dev
to install it.
After this, ensure the repositories are loaded using this command:
sudo apt-get update
Instructions to Patch, Build and Install¶
Follow these steps to patch NCURSES package on Ubuntu.
Since trunk revision XXXXX the static linking is used to build libp2j.so with integrated NCURSES calls. Please see the special notes below when static linking differs from dynamic one.
When your operating system applies software updates, if NCURSES is changed it will need to be re-patched. Please see Automating the Patching Process for a solution.
Before reading further, you should know the scripts used in the process to automate the re-patching can be used for manual patching of NCURSES 6.0, too; follow these commands to do this, without having to run each individual command described in the next sections by hand; lets assume a folder ~/setup exists:
cd ~/setup wget https://proj.goldencode.com/attachments/download/10659/patch_ncurses6.sh wget https://proj.goldencode.com/downloads/ncurses/ncurses_curses_h_in_v6.1_20200708.patch wget https://proj.goldencode.com/downloads/ncurses/ncurses_lib_getch_c_v6.1_20200708.patch chmod +x patch_ncurses6.sh sudo ./patch_ncurses6.sh $PWD
or for 5.x versions:
cd ~/setup wget https://proj.goldencode.com/attachments/download/6190/patch_ncurses.sh wget https://proj.goldencode.com/downloads/ncurses/ncurses_curses_h_in_20060828.patch wget https://proj.goldencode.com/downloads/ncurses/ncurses_lib_getch_c_v5.7_20090512.patch chmod +x patch_ncurses.sh sudo ./patch_ncurses.sh $PWD
These commands will download the patch files and run the patch_ncurses6.sh or patch_ncurses.sh script; you will need sudo rights for it.
Following sections will describe the individual commands used by the patching script.
Development Headers¶
The libncurses5-dev package is required only if your installed NCURSES version is 5.9 or earlier. Check that the libncurses5-dev package is installed. This is the developer libraries, headers and documentation for NCURSES development version 5.9 or earlier.
Use dpkg -l libncurses5-dev which should show this if it is installed:
Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=====================================-=======================-=======================-================================================================================ ii libncurses5-dev:amd64 6.0+20160213-1ubuntu1 amd64 developer's libraries for ncurses
If the text under the Version column starts with 6, then you need to remove this package using sudo apt-get remove libncurses5-dev.
Otherwise, if you are using NCURSES 5.9 or earlier, and it is not installed, install it using sudo apt-get install libncurses5-dev.
To check the NCURSES version, use dpkg -l libncurses5 for the text in the Version column.
Obtain the Source package¶
Before running this command, ensure the Source Repos are installed. A directory will be created similar to ncurses-6.1, depending upon the current available version.
cd /tmp apt-get source ncurses
Patch the Source¶
Starting in Ubuntu Jaunty Jackalope (v9.04), ncurses 5.7 is the installed version. Later Ubuntu versions use versions up to v6.2. This example will use the version numbers and patch file names for v6.
cd ncurses-6.1 wget https://proj.goldencode.com/downloads/ncurses/ncurses_curses_h_in_v6.1_20200708.patch wget https://proj.goldencode.com/downloads/ncurses/ncurses_lib_getch_c_v6.1_20200708.patch patch include/curses.h.in ncurses_curses_h_in_v6.1_20200708.patch patch ncurses/base/lib_getch.c ncurses_lib_getch_c_v6.1_20200708.patch
Or previous version:
wget https://proj.goldencode.com/downloads/ncurses/ncurses_curses_h_in_20060828.patch wget https://proj.goldencode.com/downloads/ncurses/ncurses_lib_getch_c_v5.7_20090512.patch patch include/curses.h.in ncurses_curses_h_in_20060828.patch patch ncurses/base/lib_getch.c ncurses_lib_getch_c_v5.7_20090512.patch
Build¶
./configure --libdir=/usr/lib --with-shared --with-abi-version=6 make
Install¶
The patch script automates the naming of the libncurses.so file after the make install command using:
# identify the .so name (it is version-specific)
full_libname=$(basename $(find lib/ -maxdepth 1 -type f -name "libncurses.so.*"))
major_libname="${full_libname%.*}"
# fixup libraries permissions to be accessible for building
sudo chmod o+rx /usr/lib/libform.so* /usr/lib/libmenu.so* /usr/lib/libncurses.so* /usr/lib/libpanel.so*
# fixup the main lib (the install process leaves it alone, but that won't work)
cd /lib/$(uname -i)-linux-gnu/
sudo cp $full_libname $full_libname.ori
sudo cp /usr/lib/$full_libname .
sudo rm $major_libname
sudo ln -s $full_libname $major_libname
You may need to update the below examples, should your configuration be different.
sudo make install sudo chmod o+rx /usr/lib/libform.so* /usr/lib/libmenu.so* /usr/lib/libncurses.so* /usr/lib/libpanel.so* cd /lib/x86_64-linux-gnu/ sudo cp libncurses.so.6.1 libncurses.so.6.1.ori sudo cp /usr/lib/libncurses.so.6.1 . sudo rm libncurses.so.6 sudo ln -s libncurses.so.6.1 libncurses.so.6
Cleanup¶
cd /tmp rm -fr ncurses-6.1 ncurses_*.tar.gz ncurses_*.tar.xz ncurses_*.dsc ncurses_*.asc
Known Issues¶
Distribution Customized TERMINFO¶
WARNING: if the source version you are using is not customized to your distribution it is recommended that you DO NOT run make install or make install.data (this would install a new terminfo database). It has been found that Linux distributions such as SuSE have a customized terminfo database which behaves better than the one (at least in the xterm case) which is included in NCURSES 5.5. If you do decide to install this, you really should backup /usr/share/terminfo first. However, if you use apt-get on Ubuntu, then you may safely use make install which installs everything.
Failure During FWD native Build¶
If you see the following failure during the native portion of the FWD build:
[ant:exec] gcc -o libp2j.so process.o filesys.o memory.o terminal.o library.o init.o signals.o shell.o init_linux.o process_linux.o filesys_linux.o terminal_linux.o library_linux.o signals_linux.o shell_linux.o -static-libgcc -shared -lffi -lncurses -ldl -lutil [ant:exec] makefile:247: recipe for target 'libp2j.so' failed [ant:exec] /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libncurses.a(lib_beep.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC [ant:exec] /usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libncurses.a: error adding symbols: Bad value [ant:exec] collect2: error: ld returned 1 exit status [ant:exec] make: *** [libp2j.so] Error 1 :ant-native FAILED
This is very confusing because the native build should be trying to link with libncurses.so (the dynamic library) and never with libncurses.a (the static library). The error message is poor. The problem comes from a lack of permissions to read/execute the libncurses.so.6.1 which will lead the linker to fall back to trying to link using libncurses.a. FWD's libp2j.so cannot link using libncurses.a, so the link fails. But the reason for failure is hidden.
It may be necessary to fix the permissions for the libncurses libraries in /usr/lib. Ensure these libraries can be read, by setting these permissions:
sudo chmod o+rx /usr/lib/libncurses.so.5.9 /usr/lib/libncurses.so.6.1
Full details of the problem can be seen in task #3298. The patch_ncurses.sh script included in this document has the proper chmod command that should resolve this issue. If you are using this patching script the issue should not occur.
Automating the Patching Process¶
On a production system, you can avoid changes to this package by marking the NCURSES package 'on hold', so that it will not be automatically updated. Use sudo apt-mark hold ncurses-*. Later, to remove the hold, use sudo apt-mark unhold ncurses-*. Please be advised that this is not a preferred solution since security updates would not automatically occur, possibly leading to the system being vulnerable.
The following approach will handle patching for NCURSES and also the closely related requirements for Patching TERMINFO).
The patch_ncurses.sh script was created to automate the patching and installation process for NCURSES. Just copy the script and the patch files into the same folder and run it using ./patch_ncurses.sh $PWD. The user will need sudo rights for some commands.
The patch_terminfo.sh script was created to automate the patching and installation process for TERMINFO. Just execute this using ./patch_terminfo.sh. The user will need sudo rights for some commands.
This patching will need to be repeated whenever the NCURSES package is updated. For this reason, it is best to automate this process by hooking into the apt-get update and calling the scripts whenever NCURSES is changed.
1. Place the following files in /root/:
manage_term_patches.sh
patch_ncurses.sh
patch_terminfo.sh
ncurses_curses_h_in_20060828.patch
ncurses_lib_getch_c_v5.7_20090512.patch
2. Add a file /etc/apt/apt.conf.d/95patch-ncurses with the following contents:
DPkg::Post-Invoke {"/bin/bash /root/manage_term_patches.sh"; };
3. Set the permissions as follows:
sudo chmod 0550 /root/manage_term_patches.sh sudo chmod 0550 /root/patch_ncurses.sh sudo chmod 0550 /root/patch_terminfo.sh sudo chmod 0400 /root/ncurses_curses_h_in_20060828.patch sudo chmod 0400 /root/ncurses_lib_getch_c_v5.7_20090512.patch sudo chmod 0644 /etc/apt/apt.conf.d/95patch-ncurses
Can This Patch Be Included in the Upstream NCURSES Project?¶
This is an objective the FWD project has tried to accomplish. Here was the proposal to the NCURSES maintainer Thomas Dickey:
-------- Forwarded Message --------
Subject: proposed extension API addition for ncurses 5.5
Date: Tue, 29 Aug 2006 12:30:37 -0400
From: Greg Shah ...
To: Thomas Dickey ...Mr. Dickey,
I have an application which requires that a user can generate an asynchronous notification (via CTRL-C) at any time, no matter what the state of the input or output processing of the application. This notification must be honored immediately by the application. Due to this design requirement, we have implemented our application in 2 threads.
I fully understand that NCURSES is not thread safe, but I have found a way to make it safe-enough in this particular case.
I use 1 thread for reading the keyboard via getch() and another thread for all output processing. Any time the keyboard reading thread (which is simply in an infinite loop of calling getch() with timeout set to 500 ms) happens to call into getch() at the same moment that the output thread is processing a screen update there will be a problem. In particular, getch() forces a refresh() before blocking on reading the FIFO or terminal. Since access to the internal data structures of ncurses is not protected in any way, calling a refresh() at (essentially) random times will sometimes force the output buffer to the terminal at a moment when it is in an inconsistent state (because it is in the middle of being updated by another thread). This causes unfinished escape sequences etc... to be pushed out, leaving the terminal looking corrupted.
My solution is to add a simple extension to NCURSES 5.5:
void auto_getch_refresh(bool)
This simply sets a static flag inside lib_getch.c. This flag is checked inside the wgetch_should_refresh macro to allow the disabling of refresh() during getch(). This getch_refresh flag defaults to TRUE which is completely compatible with the current NCURSES implementation. So in all current usage, there is no difference in behavior. In the case where one wishes to disable refresh during getch(), this is now possible. In particular, we found that our 2 threaded case where 1 thread is dedicated to getch() is now safe.
Some thoughts:
1. This is very low risk.
2. In the case where this new function is never called, this doesn't change the behavior in regards to standards or interface compliance.
3. The performance cost is very low (on an Intel CPU, only 3 additional instructions per getch()).
4. The memory footprint of the data is only an extra 4 bytes per process.I am hoping this will be acceptable for inclusion in the next release of NCURSES.
Thank you for your time. Please let me know if you have any questions or comments.
Greg Shah
This was his response:
-------- Forwarded Message --------
Subject: Re: proposed extension API addition for ncurses 5.5
Date: Tue, 29 Aug 2006 20:10:06 -0400 (EDT)
From: Thomas Dickey ...
To: Greg Shah ...On Tue, 29 Aug 2006, Greg Shah wrote:
My solution is to add a simple extension to NCURSES 5.5:
I'd suppose it would be more straightforward to add mutexes and measure the performance than to add a special case. What you're describing is a non-threadsafe version of one of the likely places for adding a mutex.
void auto_getch_refresh(bool)
This simply sets a static flag inside lib_getch.c. This flag is checked
inside the wgetch_should_refresh macro to allow the disabling of refresh()
during getch(). This getch_refresh flag defaults to TRUE which is completely
compatible with the current NCURSES implementation. So in all current usage,
there is no difference in behavior. In the case where one wishes to disable
refresh during getch(), this is now possible. In particular, we found that
our 2 threaded case where 1 thread is dedicated to getch() is now safe.But if two threads were to call wgetch(), that wouldn't work.
--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net
And I responded with this:
-------- Forwarded Message --------
Subject: Re: proposed extension API addition for ncurses 5.5
Date: Wed, 30 Aug 2006 07:22:19 -0400
From: Greg Shah ...
To: Thomas Dickey ...Yes, this is all true. And having multiple threads in output operations is similarly problematic. But a real thread-safe implementation is quite a large undertaking. Every shared resource and data structure would have to be protected. To do it right, a great deal of analysis is needed as to the granularity of the mutexes, possible deadlock/race conditions... and access to most data structures would best be hidden behind new internal interfaces (getters/setters or the like) to allow the mutex usage to be consistent without hard coding the mutex access everywhere. For example, even reading a flag from the WINDOW structure would require protection.
I don't have the time (does anyone?) to do a full thread-safe implementation. So my approach is a compromise to be safe-enough while minimizing the effort and risk.
Thanks,
Greg
Unfortunately, I never got any further response.
If any community member would like to take another run at this, they are welcome.
Can the NCURSES Threading Support in v5.7 Be Used?¶
We would like to eliminate the need to patch. If possible, it will take some work. Please see #2660 for the details.
Can Static Linking Help?¶
It should be possible to link libp2j.so with the static version of the patched library (libncurses.a) instead of the dynamic version (libncurses.so). The advantage is that the version installed in the OS would not need to be patched because the already patched version would be included in the libp2j.so itself. The downside would be that any security updates to ncurses would require a rebuild of libp2j.so. In my opinion, this is probably a good trade off.
Make sure to read to the end of this page (the -llibncurses.a seems to be the right approach):
https://stackoverflow.com/questions/3514852/statically-link-ncurses-to-program
© 2004-2019 Golden Code Development Corporation. ALL RIGHTS RESERVED.
patch_terminfo.sh 
(585 Bytes)
manage_term_patches.sh
(482 Bytes)
patch_ncurses.sh
(1.57 KB)
95patch-ncurses (63 Bytes)
patch_ncurses.sh
(1.77 KB)
patch_ncurses6.sh
(1.8 KB)