l2bin & l3bin-unable to read file from list.txt

By mimoyski Date 2011-03-05 12:02

hi...
somebody help, please?? i've just installed seadasVA(+auxillary data). i've downloaded MODIS level 2 and am trying to bin chl data w/ l2bin &l3bin through the GUI, but i get 'No such file or directory' error if i open from the text file list of input files. i've checked the postings here and tried their solutions but didn't work. i'm also getting a different error from theirs. below are details of my processing:

working environment: winXP-32,seadasVA6.2
   full permission on /home/seadas -recursive

input files: 1km l2binned (binned individually because l2bin doesn't work from
textfile input file list!)

contents of 'list.txt' input for modis l3bin:(i used 2 testfiles)
/home/seadas/shared/A2010157044000.L2_LAC_OC.x.hdf.L3b_GAC
/home/seadas/shared/A2010159060500.L2_LAC_OC.x.hdf.L3b_GAC

what i've tried:
1) put all the files in 1 folder (/home/seadas/shared, & /home/seadas, & /mnt/hgfs/seadas_shared). i change the paths in the list.txt when i tried these different
folders.
--can read filename from list.txt, populate the first few details in the l3bin GUI,
   BUT CANNOT access/open the file. i get 'No such file or directory' and 'unable
   to open file'. and i get this processing log:

IDL Version 7.0 (linux x86 m32). (c) 2007, ITT Visual Information Solutions

% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
SeaDAS Version 6.2 (pid = 3458)

grp_name = Level-3 Binned Data.

scriptfile : /tmp/SDS_L3BIN.scr
IDL Version 7.0 (linux x86 m32). (c) 2007, ITT Visual Information Solutions

% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
SeaDAS Version 6.2 (pid = 3544)

grp_name = Level-3 Binned Data.

grp_name = Level-3 Binned Data.

l3bin.pro: cmd_final =
/bin/bash -c "l3bin IN=/mnt/hgfs/seadas_shared/list.txt OUT=/home/seadas/M20101572010159.L3b_GAC latsouth=10.0000 latnorth=30.0000 lonwest=105.000 loneast=130.000 noext=1 OUT_PARM='aot_869:angstrom:Rrs_412:Rrs_443:Rrs_469:Rrs_488:Rrs_531:Rrs_547:Rrs_555:Rrs_645:Rrs_667:Rrs_678:chlor_a:Kd_490:ipar:nflh:par'"
L3BIN 3.27 (Dec 23 2010 12:16:26)
2 input files
0 /home/seadas/shared/A2010157044000.L2_LAC_OC.x.hdf.L3b_GAC
HDF5-DIAG: Error detected in HDF5 (1.8.5-patch1) thread 0:
#000: H5F.c line 789 in H5Fis_hdf5(): unable to open file
major: Low-level I/O
minor: Unable to initialize object
#001: H5FD.c line 1088 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#002: H5FDsec2.c line 370 in H5FD_sec2_open(): unable to open file: name = '/h', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
major: File accessability
minor: Unable to open file

exit_status= 0

2) tried to load and display each file manually to make sure the files can be read
and not corrupted.
--the files displayed properly, meaning the input files were not corrupted.

with all the files having full permissions, and input files not corrupted, i don't really know what i'm missing. and the thing is, the only way (at least that i know of) to input files in l3bin is through the textfile list.
would really appreciate if someone can help please, please???
thanks...

mimoy

Attachment: list.txt (0.1k)

By WhiteG Date 2011-03-06 23:08

The l3bin program is fussy about the format of the list file. I've seen similar HDF5-DIAG messages when the list file had a blank line.
It appears to getting only the leading "/h" of the filename. Try examining the file with a different editor. You can also use "cat -v list.txt"
to have non-printing characters displayed. Is the file 118 characters long?

By mimoyski Date 2011-03-08 14:17

hi george

thanks for responding but i'm afraid it didn't work :-(

tried the "cat -v list.txt" and a couple of other editors,
there were no non-printing characters in the file and doesn't reach 118 characters. and after trying a
couple of times getting the same error, it's gotten worse!!! it's not even reading the list at all and i'm
getting this warning dialog box:
----------------------------------------------------------------------------------------
Warning
! The following error was encountered: HDF_SD_END: Unable to close the HDF-SD interface
(SD_ID=393216). Please consult supplier of application.
----------------------------------------------------------------------------------------

this was the same error message i was getting with the l2bin earlier, but since i was able to work
around it by processing the images individually, i didn't mention it anymore. but now it's happening
with l3bin as well--help!!!

any suggestions or ideas why all these are happening? is there something wrong with seadasVA,
or something i have to configure to fix this? is it possible it could be some compatibility issue
with my laptop? (it's a DELL vostro).

by the way, i've also attached the "list.txt" (in my first post), in case you or anybody would want
to check.
i only have windows platform available so i'm really counting on seadasVA. please, i'd really appreciate
any help on this--from you (or anybody who might also have any suggestions(?) ) thanks!!!

mimoy

By WhiteG Date 2011-03-08 17:01

I doubt there is a compatibility issue, but strange problems are often a result of resource limitations.

1. how much free space is available on the Windows disk? I like to have at least 15% available, but
more is better.
2. how much RAM is allocated to the SeaDAS VA? on a 2G system I would try 720, on a 3G
system you could go to 1G.
3. how much disk space is available to linux?   You can display this using "df -m" -- look at the
entries for /dev/sda1 /dev/sdb1 and .host

VMware warns that AV software can interfere with virtual disks. Can you exclude the virtual disks
from the AV scanner?

You should check the logs for error messages -- if there is an experienced linux user around they
can help you find and decipher the linux logs.   Windows has the "event viewer" which is also
worth examining for signs of trouble.   Virtual machines are handy, but they can't function well
if the host system is not working properly.

It is good policy to run the seadas benchmark script to make sure things are working and also
to check for a performance problem.   The whole script should run in under 20 minutes -- on a
newer laptop it runs in about 10 mins.

By mimoyski Date 2011-03-11 10:00

you were right, george!!!
it's resources, i'm out of disk space (i assume it's linux disk space-??). i ran the benchmark and
got a "No disk space..." message after Level-2 generation. df -m gave me these:
Filesystem    1M-blocks Used Available Use% Mounted on
/dev/sda1    7669 7280 1 100% /
none    498    1    497   1% /dev
none    502    1    502   1% /dev/shm
none    502    1    502   1% /var/run
none    502    0    502   0% /var/lock
none    502    0    502   0% /lib/init/rw
.host:/    102587   40953 61635 40% /mnt/hgfs

i was able to increase RAM settings to 1024 and increase virtual disk space from 8G to 21G. there's
an option to do this in "VMR>edit virtual machine settings>hardware tab>hard disk(SCSI)>utilities>
expand". The changes were reflected on the settings gui plus a message:
---------------------------------------------------------------------------------------------
"The disk was successfully expanded. You must repartition the disk and expand the file systems from within the guest operating system.
Disk information:
Disk space is not preallocated for this hard disk. Hard disk contents are stored in a
single file.
Capacity:
Current size: 3.5 GB
System free: 72.4 GB
Maximum size: 21 GB "
---------------------------------------------------------------------------------------------
sure enough,i checked with dm -f, the changes didn't take effect, the numbers didn't change.

so, at least this looks good, problem identified (i think!). done initial steps to expand virtual
disk space (hope i did the right thing). i guess next step is repartition and expand file systems
to match the virtual disk space changes--no idea how to do this! could you please help me with instructions on how to do this?

thanks so much!!!
mimoy :-)

By WhiteG Date 2011-03-11 12:34 Edited 2011-04-01 15:34

SeaDAS 6.2 VA provides only a single 8G virtual disk. Perhaps it should provide 2 virtual disks. In Windows, virtual disks are large files: SeaDAS_6.2-000001-cl1.vmdk (/dev/sda1 in linux, mounted on /). My PC also has SeaDAS_6.2-d2.vmdk (/dev/sdb1 in linux, mounted on /home). In the "Virtual Machine Settings" hardware tab these are shown as Hard Disk (SCSI) 8GB and Hard Disk 2 (SCSI) 20GB.

Moving the /home directory to a new virtual disk is a bit tricky. The safest way to do it is to install a 2nd virtual machine, mount both
the original and new seadas virtual disks, move the /home folder to the new disk, edit the file <slash>etc<slash>fstab (the forum does not like real / here!) should to add a line of the form:

/dev/sdb1 Xhome ext4 errors=remount-ro 0 0

The problem for you is that you now have a lot of data on the first virtual disk that should be in the 2nd virtual disk. Experienced linux users generally run a "df" command at least once a day to make sure all the disks are available and that none are getting too full, but experience does not come without pain.

By mimoyski Date 2011-03-14 12:35

ok, to answer your questions first...
1) extracted seadasva62.exe contains only 1 .vmdk (SeaDAS_6.2-000001-cl1.vmdk), and
SeaDAS_6.2.0.vmx. i thought this was ok as indicated in the seadasva instruction page. just to
make sure, i did a fresh download again and extracted--same,contained only those 2 files.

2) Hard Disk 2 does not exist in the Hardware tab. only an 8GB(max) exists linked to
SeaDAS_6.2-000001-cl1.vmdk

3) in fstab, this line does not exist:
(/dev/sdb1 Xhome ext4 errors=remount-ro 0 0)

after checking all these, i made a fresh install (VMR,seadasva62), from fresh downloads.what i got:
1) extracted seadasva62.exe contained the same 2 files as above.

2) Hard Disk 2 still doesn't exist in Hardware tab. I tried to "Add" a Hard disk but since there was
only 1 .vmdk file, i "created" another one, 20GB size and named it SeaDAS_6.2-d2.vmdk. It
showed in the Hardware tab as pre-allocated hard disk (SCSI) 20 GB.

3) in fstab, the "/dev/sdb1..." line still doesn't exist.

sorry for doing the fresh install. i thought it could give more info, and i did get the infos from
the old installation before i got rid of the it anyway. i don't mind "redoing from the start" for
'n' times--no pain, no gain!!! as long as this can be figured out to have seadasva running and help others who may be having the same issue.

thanks. will wait for your instructions.

mimoy :-)

Attachment: fstab (0.9k)

By WhiteG Date 2011-03-14 18:32 Edited 2011-04-01 15:27

8GB is pretty tight as SeaDAS with modis and seawifs processing and sources and a benchmark run uses around 4GB over and above the basic OS (I guess a couple GB). The downloads are around 3GB. With care you should be able to install in 8GB if you keep the .tar.gz files and data in the shared folder (e.g., on a Windows disk), but if you download into the SeaDAS VM then it isn't surprising that you run out of space.

Having created a 20GB virtual disk you need to format it and then mount it. You can find tutorials on the web, but many of these will use a modern GUI such as gparted that probably isn't installed in the SeaDAS VA, so you need to look for the command-line method or install gparted.

If you don't find a way to format and mount the 20GB virtual disk mounted then you could download the .tar.gz files to the Windows shared folder or to an external USB drive, but keep a careful watch on the free space with df.

By White.Jason Date 2011-04-07 16:43

WhiteG,

I am also running having problems running out of space on my virtual machine while processing modis data. I decided to re-download SeaDas 6.2 and download all the processing files directly into my shared folder. I then went in and untarred all my processing files via the SeaDas terminal window. Everything looks to be set-up correctly, however I'm having problems setting my default working correctly so processing proceeds correctly off my shared folder.

I see the default working directory is /home/seadas
My shared folder currently exist in this structure /home/seadas/Desktop/SeaDas_Shared/seadas_shared
So under the default working directory GUI, I changed the drop down menu to /mnt/hgfs

I have then tried making my default working directory similar to default using my shared folder.
EX: /mnt/hgfs/seadas_shared or /mnt/hgfs or /mnt/hgfs/seadas_shared/run

Currently when I even try to updated my aqua and terra calibration files, that doesn't even work. So I know I have an issue with the default processing directory.

Any advice is greatly appreciated,
Jason

By @sean Date 2011-04-08 17:48

Jason,

The latest update increases the size of the virtual machine disk.

Regards,
Sean

By mimoyski Date 2011-04-10 20:41

hi WhiteG and sean...
sorry for this late update, just got back from fieldwork.the recent seadasva6.2 update was good news, but still having problems opening the hdf's.i'll pick up where i've left off with WhiteG to explain clearly.
after following WhiteG's advice (thanks much, by the way :-)

),my disk resource issue with the unupdated seadasva6.2 was solved, i was able to run benchmark, which i took as a good sign. i didn't get to the virtual disk formatting anymore.

then update_02 was released. so i made fresh downloads of seadasva6.2,auxillary files and the update, then installed. the 120G virtual disk was detected, benchmark test was ok, virtual disk usage (df -m) was ok--all good signs. but when i run l3bin this is the processing error log that i get:
------------------------------------------------------------------------------------
l3bin.pro: cmd_final =
/bin/bash -c "l3bin IN=/mnt/hgfs/seadas_shared/list2.txt OUT=/home/seadas/shared/M20101592010157.L3b_GAC latsouth=10.0000 latnorth=30.0000 lonwest=110.000 loneast=130.000 noext=1 OUT_PARM='aot_869:angstrom:Rrs_412:Rrs_443:Rrs_469:Rrs_488:Rrs_531:Rrs_547:Rrs_555:Rrs_645:Rrs_667:Rrs_678:chlor_a:Kd_490:ipar:nflh:par'"
L3BIN 3.27 (Apr 7 2011 14:39:19)
2 input files
0 /home/seadas/shared/A2010159060500.L2_LAC_OC.x.hdf.L3b
1 /home/seadas/shared/A2010157061500.L2_LAC_OC.x.hdf.L3
HDF5-DIAG: Error detected in HDF5 (1.8.5-patch1) thread 0:
#000: H5F.c line 789 in H5Fis_hdf5(): unable to open file
major: Low-level I/O
minor: Unable to initialize object
#001: H5FD.c line 1088 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#002: H5FDsec2.c line 370 in H5FD_sec2_open(): unable to open file: name = '/home/seadas/shared/A2010157061500.L2_LAC_OC.x.hdf.L3', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
major: File accessability
minor: Unable to open file

exit_status= 0
----------------------------------------------------------------------------------
i've checked the input file (list2.txt) for non-printing characters--it's ok. list2.txt contains 2 lines, each for the path of two l2binned files.from the error log,it seems that the last character in the 2nd line cannot be read. i tried with another list.txt file containing only 1 line, and it did the same--the last character 'b' was chunked out from '.L3b'. i guess that's why the file couldn't be opened, it's always missing the last character.

would appreciate any advice/instructions you could give, please?

thanks!!!

By @sean Date 2011-04-11 13:40

It seems as if the file does not have the correct line termination.

How are you creating the input list file?

Sean

By WhiteG Date 2011-04-11 14:41

Sean is probably on the right track -- l3bin is very fussy about the format of the list file. Text files in unix/linux normally have a single "newline" character at the end of each line, while Windows uses two characters. Some editors don't always put the newline character at the end of the last line of a file. Here is an example of a file that is missing the trailing newline character:

$ echo -en A2006167181000.L3b_space.hdf\\nA2006167181000.L3b_space.hdf > files.list
$ cat files.list
A2006167181000.L3b_space.hdf
A2006167181000.L3b_space.hdf$

Note that the prompt ($) appears on the same line as the 2nd filename. In the vi editor I see:

A2006167181000.L3b_space.hdf
A2006167181000.L3b_space.hdf
~
[...]
~
"files.list" [noeol] 2L, 57C

The [noeol] is a warning that the last line does not have an "end of line" character.

By mimoyski Date 2011-04-16 10:19

hi whiteG and sean...

it's working now!!!THANKS SO MUCH!!
i've been using midnight commander and vi to check non-printing characters. i didn't change text editors but just followed your advice to check again-nothing to lose anyway. well, for some reason it worked this time :-)

another error popped up though. the generated l2 and l3-binned files generated cannot be displayed. i'm getting a segmentation fault error and the seadas gui closes.
i made a new post for this so others who might have the same problem can track it down. please reply to that post if you could give some advice. thanks again!!!

mimoyski

By WhiteG Date 2011-04-16 10:49

I'm getting the segfault displaying binned files. You should be able to use the command-line smigen to create a mapped image that can be loaded in SeaDAS. An example
from the benchmark script is:

smigen ifile=A2006167181000.L3b_time.hdf \
   ofile=A2006167181000.smi_chlor_a_mn.hdf \
   prod="chlor_a" resolution=2km \
   loneast=-62 lonwest=-78 latsouth=30 latnorth=46

By mimoyski Date 2011-04-16 16:07

thanks WhiteG !!! :-)

the problem was solved by the latest seadasva_6.2 update_02a. but i did take a
note of your workaround--a backup solution in future, just in case.

By WhiteG Date 2011-04-17 12:45

Good to know update02a is working. I've learned to record fixes and workarounds in the forums so I (and others!) can find them when I need them in the distant future (and can be found using google search "site:http://oceancolor.gsfc.nasa.gov ...").