FWD » Core Development » Base Language

With server-side resources, all paths are relative to the FWD server's JVM working dir, and not the working directory specified in directory.xml for that FWD account.

We need to use this folder as the first one when doing propath, file-info:file-name, icon, etc lookups.

Artur, do you want to work on this?

Constantin Asofiei wrote:

Artur, do you want to work on this?

Sure.

Yes, but don't rely on '.' to be first in the path. Any relative path should resolve to workingDir first, and after that other places (although here there may be a security issue - do we allow JVM working dir resolution of files?).

legacy-system/absolute-path-prefix-map is used to re-map absolute paths, so I can't tell how easy is to use that support.

Artur, please check in initFileInfo that the given name is not an absolute name first. If this is the case, then working dir should not be looked. You can end up with cases when the name is /foo/bar and the working dir also has /path/to/workDir/foo/bar, while it exists on disk as /foo/bar.

Also, the change in getWorkingDirectory is not correct. When server-side filesystem is not used, this needs to return the working dir for the FWD Client JVM. Keep in mind that clientConfig/workingDir is optional.

Constantin Asofiei wrote:

Artur, please check in initFileInfo that the given name is not an absolute name first. If this is the case, then working dir should not be looked. You can end up with cases when the name is /foo/bar and the working dir also has /path/to/workDir/foo/bar, while it exists on disk as /foo/bar.

Yes, this is a compromise, I am not sure how to check if a path is absolute, like you mentioned, a path can be both absolute and relative. In initFileInfo we only have the path string.

Can't File.isAbsolute work?

It does :) I also found now that Paths.get(name).isAbsolute() works, I am wrapping up the changes.

• getWorkingDirectory().value.concat("/").concat(name); - does this work on Windows? Don't we need to append OS path separator?
• wa.info = wa.fs.accessFileInfo(origName); - this doesn't consider if origName is null (previously it did)

Also, we need to test the impact on customer projects where server-side filesystem is used.

I addressed the review.
Constantin, please email me the testing that has to be done.

Only the large gui application has been tested, I have not seen server side fs being used in other projects.

Question: did you test the propath lookup, also? Usually there are a '.' or relative paths there, and this means 'current working dir'.

I only used the tests in note 8.

Do you mean something like this:

PROPATH = ENTRY(1,PROPATH) + "/prop".

FILE-INFO:FILE-NAME = "test.txt".

IF FILE-INFO:FULL-PATHNAME <> ? THEN
  MESSAGE "Test 1: OK - found:" FILE-INFO:FULL-PATHNAME
    VIEW-AS ALERT-BOX.
ELSE
  MESSAGE "Test 1: FAIL - file not found" 
    VIEW-AS ALERT-BOX ERROR.

test.txt exists both in prop and working dir defined by the user, the file should be found in the working dir, because it has the priority.

I mean more like setting an icon for a button (for which the location uses PROPATH AFAIK), or using a COPY-LOB (with files as source/target) or even a INPUT FROM or OUTPUT TO statement. All these can work with the server-side file-system, and they must be relative to the agent's workingDir in PASOE, if the path is relative.

Greg: in terms of #9055, I think there is more than what we can identify easily. We can release 10313a, and after that figure out all features which require file-system access.

message CURRENT-WINDOW:LOAD-ICON("icon.ico").
message CURRENT-WINDOW:LOAD-ICON("./icon.ico").
message CURRENT-WINDOW:LOAD-ICON("pkg/icon.ico").
message CURRENT-WINDOW:LOAD-ICON("./pkg/icon.ico").

DEFINE VARIABLE dInFile    AS CHAR  NO-UNDO INIT "./test.txt".  
DEFINE VARIABLE dataMemptr AS MEMPTR NO-UNDO. 
COPY-LOB FROM FILE dInFile TO dataMemptr. 

DEFINE VARIABLE dInFile2    AS CHAR  NO-UNDO INIT "test.txt".  
DEFINE VARIABLE dataMemptr2 AS MEMPTR NO-UNDO. 
COPY-LOB FROM FILE dInFile2 TO dataMemptr2. 

DEFINE VARIABLE dInFile3    AS CHAR  NO-UNDO INIT "pkg/test.txt".  
DEFINE VARIABLE dataMemptr3 AS MEMPTR NO-UNDO. 
COPY-LOB FROM FILE dInFile3 TO dataMemptr3.

DEFINE VARIABLE dInFile4    AS CHAR  NO-UNDO INIT "./pkg/test.txt".  
DEFINE VARIABLE dataMemptr4 AS MEMPTR NO-UNDO. 
COPY-LOB FROM FILE dInFile4 TO dataMemptr4.

INPUT FROM VALUE("test.txt").
INPUT CLOSE.

INPUT FROM VALUE("./test.txt").
INPUT CLOSE.

INPUT FROM VALUE("pkg/test.txt").
INPUT CLOSE.

INPUT FROM VALUE("./pkg/test.txt").
INPUT CLOSE.

OUTPUT TO VALUE("testOut.txt").
INPUT CLOSE.

OUTPUT TO VALUE("./testOut2.txt").
INPUT CLOSE.

OUTPUT TO VALUE("pkg/testOut.txt").
INPUT CLOSE.

OUTPUT TO VALUE("./pkg/testOut2.txt").
INPUT CLOSE.

The tests fail because the methods do not use initFileInfo.

Constantin, do we need to go through all the methods and functions that could take a relative path as an argument and add workig dir resolving to them?

Artur Școlnic wrote:

Constantin, do we need to go through all the methods and functions that could take a relative path as an argument and add workig dir resolving to them?

Yes, make a list of the statements we need to test. There is also COPY-LOB TO file

Isn't there a place where all relative file names are resolved to the current working dir? Without server-side fs, all these methods do work correctly, they are resolving the path to the actual current working dir.

There is TargetData and SourceData, which in the end rely on StreamFactory.openFileStream

For icons, there is TC.loadWindowIcon and loadImage, which rely on searchPathWithExts and searchPath in FileSystemDaemon.

I can't tell what other location would be, look for references of FileInputStream and other APIs which load from file-system.

I was hoping for a single method or class where all the lookups are performed :)

Eventually any relative path should be prefixed with the OS working dir or PROPATH. Maybe we can just add an entry to the PROPATH (at the front) with the client working dir.

It's a long shot, but:

cd ../work/ && ../client/client.sh

Effectively sets the user.dir to any path we choose, thus simulating a custom working dir :)

Artur Școlnic wrote:

It's a long shot, but:
[...]

Effectively sets the user.dir to any path we choose, thus simulating a custom working dir :)

Well, you can't do that in PASOE :) there is no FWD client there, just the FWD server JVM. And with server-side file-system, the 'user.dir' is the FWD Server's working dir.

Is the PROPATH always used in relative path resolving? If so, can't we just add the client working dir in the front of the PROPATH, thus all relative paths will be resolved using it first?

Artur Școlnic wrote:

Is the PROPATH always used in relative path resolving? If so, can't we just add the client working dir in the front of the PROPATH, thus all relative paths will be resolved using it first?

You can do that in searchPath and searchPathWithExts, but not change the actual PROPATH - this can be interpreted/managed by the application. And also this can be done only if you have i.e. "." or other relative paths in the PROPATH. You can't just change everything. And also keep in mind the entries in the PROPATH are also paths relative to name_map.xml, so not everything is a OS relative path.

Better said: when doing a PROPATH lookup, if the entry is a relative path, append the working dir for that context, for server-side filesystem.

Greg: in terms of #9055, I think there is more than what we can identify easily. We can release 10313a, and after that figure out all features which require file-system access.

I've OK with merging before all cases are worked so that the customer can do some testing. My sense is that these are very common features and the resulting system will be broken until all of the relative server-side file system usage is fixed.

SO: yes, merge quickly but no, don't defer the work.

Artur Școlnic wrote:

Isn't there a place where all relative file names are resolved to the current working dir? Without server-side fs, all these methods do work correctly, they are resolving the path to the actual current working dir.

I think we need to create such a thing and make it used everywhere. On a client-side process, it would use the actual current-directory the way it does now.

Extended the support for load-icon, copy-lob, input/output in 10313a/16068. Now the test cases from note 25 are passing.

Greg Shah wrote:

I think we need to create such a thing and make it used everywhere. On a client-side process, it would use the actual current-directory the way it does now.

My latest changes and the ones I plan to make here, are using FileSystemOps.searchPath everywhere a path is used to find or create resources, I think this method is a good candidate for a single point of entry for path look-ups throughout the project. Since it seems like we will have to scan the project and add support for working dir resolving, might as well do it using the same method.

• searchPath
  • will return the same relative path, not absolute. I don't know if that will help.
  • searchPath - this if (wa.serverSideFs && !Paths.get(filename).isAbsolute()) will append working dir even if that is a convert program name. What will SEARCH("a.p") return?
• searchResourceJars - what if extensions is not null?

Focus on the SEARCH function and other non-UI statements for now please.

Constantin Asofiei wrote:

• searchPath
  • will return the same relative path, not absolute.

Do you mean in non server-side fs?

• What will SEARCH("a.p") return?

It will return the full path relative to the working dir, if it is in the working dir.

Constantin Asofiei wrote:

• searchResourceJars - what if extensions is not null?

If the filename does not have the extension, the file will be resolved to the working dir, if it does have an extension and extensions is not null, the method will fail to find the file, same as before the changes, I think.

• in FileSystemOps.initFileInfo - why origName.isEmpty()?
  

        // If the first lookup failed, try again with the user.dir
        if(origName != null && origName.isEmpty() && (wa.info == null || wa.info.pathname == null))
        {
           wa.info = wa.fs.accessFileInfo(origName);
        }
  

• in FileSystemOps.searchPath - filename.startsWith("./") is not OS-independent
• searchResourceJars - that always uses relative filenames (I think to root package); this code can't be here:
  

        WorkArea wa = work.obtain();
        if (wa.serverSideFs && !Paths.get(filename).isAbsolute())
        {
           filename = getWorkingDirectory().value.concat(getFileSeparator()).concat(filename);
        }
  
  

• FileSystemDaemon.searchPathWithExts - the javadoc says that this also does PROPATH lookups - but I can't find where this would be done. If the javadoc is right, doing filename = getWorkingDirectory().concat(getFileSeparator()).concat(filename); will not allow looking up paths in the propath.

Constantin Asofiei wrote:

• FileSystemDaemon.searchPathWithExts - the javadoc says that this also does PROPATH lookups - but I can't find where this would be done.

    * Search the current directory (and PROPATH if it is configured) for a given filename
    * with the specified extensions. 

I included this change here because the javadoc states that the search is done against the current dir first, in my mind that is the user.dir, so for server-side it should be the defined working dir. The propath lookup is a secondary choice, and remains so after the changes.

I addressed the issues from note 42. I tested the customer project with 9055, 10313a an PASOE, harness and webui navigation was ok.

Constantin, is there something else we should address at this point?

Artur Școlnic wrote:

Constantin, is there something else we should address at this point?

I think if we fix TargetLob the append working dir to the filename if this is a relative path, we are OK.

I added these tests to the suite:

DEFINE VARIABLE dOutFile AS CHAR  NO-UNDO INIT "./test_out.txt".  
COPY-LOB FROM FILE dInFile TO FILE dOutFile. 

DEFINE VARIABLE dOutFile2 AS CHAR  NO-UNDO INIT "test_out2.txt". 
COPY-LOB FROM FILE dInFile2 TO FILE dOutFile2. 

DEFINE VARIABLE dOutFile3 AS CHAR  NO-UNDO INIT "pkg/test_out.txt".  
COPY-LOB FROM FILE dInFile3 TO FILE dOutFile3.

DEFINE VARIABLE dOutFile4 AS CHAR  NO-UNDO INIT "./pkg/test_out2.txt".
COPY-LOB FROM FILE dInFile4 TO FILE dOutFile4.

Are they created in the working dir of the FWD client, when server-side resources are present?

Constantin Asofiei wrote:

Are they created in the working dir of the FWD client, when server-side resources are present?

Yes.
On another note:

wa.serverSideFs = OSResourceManager.getInstance().isServerSide(OsResourceType.FILESYSTEM);

Artur Școlnic wrote:

Constantin Asofiei wrote:

Are they created in the working dir of the FWD client, when server-side resources are present?

Yes.
On another note:
[...]
is true even if i don't have the server-side enabled in the directory, is that expected?

If you are running in full PASOE, then yes.

I don't think I am , it is a dataset project with a standard directory. What is a sure way to check?

Assuming that the wa.serverSideFs is set correctly, the changes are fine. Constantin, I already tested a customer project with full pasoe and 10313a on linux, I intend to test on windows also, is there something else that should be tested?

There is an issue with the windows project

java.lang.NullPointerException
    at com.goldencode.p2j.ui.chui.ThinClient.switchWorker(ThinClient.java:20785)
    at com.goldencode.p2j.ui.chui.ThinClient.redirectOutput(ThinClient.java:20446)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)

It was a directory issue. With full PASOE on both linux and windows 10313a passed the testing. There is a minor problem in a customer project, but it is present in trunk also, we should defer it to another task.

Artur Școlnic wrote:

There is a minor problem in a customer project, but it is present in trunk

On every other 5-7 login a method is invoked which is not in the procedure (.p file) in the handle passed as argument to the invoke.
this is the stack trace

at com.goldencode.p2j.util.ControlFlowOps$InternalResolver.resolve(ControlFlowOps.java:11441)
at com.goldencode.p2j.util.ControlFlowOps$Resolver.resolve(ControlFlowOps.java:11416)
at com.goldencode.p2j.util.ControlFlowOps.invokeImpl(ControlFlowOps.java:7426)
at com.goldencode.p2j.util.ControlFlowOps.invoke(ControlFlowOps.java:4576)
at com.goldencode.p2j.util.appserver.CoreAppserver.invokeImpl(CoreAppserver.java:628)
at com.goldencode.p2j.util.appserver.CoreAppserver.lambda$invoke$2(CoreAppserver.java:281)
at com.goldencode.p2j.util.appserver.CoreAppserver$$Lambda$2131/0x000078bc018ad768.get(Unknown Source:-1)
at com.goldencode.p2j.util.appserver.CoreAppserver.executeScoped(CoreAppserver.java:1692)
at com.goldencode.p2j.util.appserver.CoreAppserver.invoke(CoreAppserver.java:279)
at com.goldencode.p2j.util.appserver.MultiSessionAppserverManager.lambda$invoke$13(MultiSessionAppserverManager.java:1400)
at com.goldencode.p2j.util.appserver.MultiSessionAppserverManager$$Lambda$1789/0x000078bc0183dde0.run(Unknown Source:-1)
at com.goldencode.p2j.util.appserver.MultiSessionAppserver$3.coreRun(MultiSessionAppserver.java:793)
at com.goldencode.p2j.util.appserver.MultiSessionAppserver$MsaTask.run(MultiSessionAppserver.java:2654)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:840)
at com.goldencode.p2j.util.appserver.MultiSessionAppserver$MsaThread.run(MultiSessionAppserver.java:2417)

It is not quite clear to me where does the method name and procedure handle come from, adding to the complexity of the problem is the non deterministic nature.
Using the classic mode, this call does not happen at all.

Constantin, is there a chance that with PASOE, new pieces of code are being accessed in the customer project and the invoke is simply incorrect?

Please create a task in the customer's project to discuss this.

10313a will not be delayed for this bug.

I tested harness+webui navigation on linux and windows, it passed.
I also asked my colleagues to test the branch with other projects, we should have the results tomorrow.

A regression was found on another customer app. Testcase:

define var existentFile as char initial "/dev/null". 

FILE-INFO:FILE-NAME = existentFile.
FILE-INFO:FILE-NAME = "nonExistentFile".

message FILE-INFO:FULL-PATHNAME. // should be ?, but is /dev/null

1. // try to look up the file using the user defined working dir first
   // when server side file system is on and the path is relative
   if (wa.serverSideFs && !Paths.get(name).isAbsolute())
   {
      name = getWorkingDirectory().value.concat(getFileSeparator()).concat(name);
      wa.info = wa.fs.accessFileInfo(name);
   }

   
   wa.serverSideFs is false.

2. // If the first lookup failed, try again with the user.dir
   if(origName != null && !origName.isEmpty() && (wa.info == null || wa.info.pathname == null))
   {
      wa.info = wa.fs.accessFileInfo(origName);
   }

   
   wa.info.pathname contains the path from the first FILE-INFO:FILE-NAME.

Thank you for the testing and the test case, Razvan.
Constantin, I added a check for the previously computed file info name, looks like in some cases the pathname is stored and messes up the current call.
We will continue with the testing.

The latest revision solved the original issue, but now I've detected another regression related to another changes. Please check this testcase:

define var windowsStyleAbsolutePath as char initial "\home\test.csv".
define stream testStream.

OUTPUT STREAM testStream TO VALUE(windowsStyleAbsolutePath).
OUTPUT STREAM testStream CLOSE.

You added in StreamFactory.openFileStream():

if (filename != null && !Paths.get(filename).isAbsolute())
{
   if (write)
   {
      filename = FileSystemOps.getWorkingDirectory().value.concat(getFileSeparator()).concat(filename);
   }
   else
   {
      filename = FileSystemOps.searchPath(filename).toStringMessage();
   }
}

The path is absolute, but is not UNIX-style. The resolving of this path will be made on the client later, through FileSystemDaemon.openFileStream with resolvePathName which will switch the path separators with File.separator, or /. As such, the code will add an unwanted working directory to the path.

It looks like I forgot to enable the working dir resolving only for server-size fs, I fixed that and asked Razvan to test again.

No regressions found with rev. 16131.

Thank you, Razvan.

Large GUI app unittests passed!

Chui regression tests passed.
FwdTests passed.
Constantin, I think we can merge.

10313a was merged as rev 16125.

• PROPATH: used to find ABL code and classes exclusively.
• Working directory (CWD / workDir): used to resolve relative file paths exclusively.
• There isn't a backup lookup policy, if a procedure is not found in the PROPATH or a file in the workDir, an error is raised.

Both properties are set in openedge.properties

[AppServer.Agent]
    workDir=${CATALINA_BASE}/work
    ...
    PROPATH=${CATALINA_BASE}/openedge,${DLC}/tty,${DLC}/tty/netlib/OpenEdge.Net.pl

Unless there are no more concerns, I will start testing.

Constantin, are we ok to merge this one?

10313b is in trunk/16464.