out_track in command mode

By WhiteG Date 2009-12-15 12:43

As it happens, I'm busy doing much the same sort of data extractions this week.

Your script is written as a bash shell script, but you want to run it using "seadas -b my_script2" (after removing the
hash-bang line).   The errors you get are because bash doesn't understand the seadas commands in the script.
You might want to read a bit on the <linuxcommand.org> site (also very useful for Mac OS X users) so you
understand how shell scripts work.

You only need two commands (load and out_track):
   -----------<8
pgm='<name of script>'
data_fnm='<name of data file>'
data_fty='<file type>'
trk_fnm='<name of track file>'
out_fnm='<name for output file>'
fnm = file_search(data_fnm, /EXPAND_TILDE, /TEST_READ)
if (fnm ne "" ) then load, fnm, ftype = data_fty $
else message, pgm+': File <'+data_fnm+'> could not be read.'

fnm = file_search(trk_fnm, /EXPAND_TILDE, /TEST_READ)
if (fnm ne "" ) then $
   out_track, iband=1, ifile=fnm, skip=1, ofile=out_fnm, /no_connect $
else message, pgm+': File <'+trk_fnm+'> could not be not read.'
-----------<8

In practice, a bash script may be used to create a seadas script with
the filenames set and then run the generated script with seadas.
This approach is useful if you are processing 100's of files and
don't have an IDL license, because you can use a loop in a shell
script to process lots of files with one command. If you have IDL,
you can write the loop in IDL.

By mike Date 2009-12-15 15:52

Great, thanks George. And there's also a FAQ that explains things.

By dash Date 2009-12-15 16:36

Thanks George and Mike. Sorry to say that I am still struggling on trying to set this program right.

I created a .dat file with a pair of lat/lon and named it as: test_track.dat (attached).
As per George's post, I created a program file and named it as: test_modis_extract (attached).
Moved these two files to benchmarks folder in SeaDAS VA to see if I am able to extract pixel values from the file, A2006167181000.L2 for just one location.
I am getting this error message, upon running it as follows:

seadas@seadasva:~/shared/benchmarks$ seadas -b test_modis_extract
IDL Version 7.0 (linux x86 m32). (c) 2007, ITT Visual Information Solutions

% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
SeaDAS Version 6.0 (pid = 7398)
% HDF_SD_START: Invalid HDF file or filename ((null)).
[MAIN]: Error executing the command:
if (fnm ne "" ) then load, fnm, ftype = data_fty else message, pgm+': File <'+data_fnm+'> could not be read.'
MAIN: Error executing command in batch file
seadas@seadasva:~/shared/benchmarks$

I really appreciate your help.
regards,
Dash

Attachment: test_track.dat (0.0k)

Attachment: test_modis_extract (0.5k)

By mike Date 2009-12-15 17:27

Ok you almost had it Dash.. I've pasted the output below with changes in bold. The problems were that since fnm is a string array you need to specify the first element, and that since you only have one line in your dat file, you need to use skip=0 for the out_track command. Remember that to troubleshoot bugs in a program you can step through the program one line at a time using print statements to narrow down the problem.

pgm='test_modis_extract' data_fnm='A2006167181000.L2' data_fty='MODIS' prod='chlor_a' trk_fnm='test_track.dat' out_fnm='test_output.txt' fnm = file_search(data_fnm, /EXPAND_TILDE, /TEST_READ) if (fnm ne "" ) then load, fnm(0), ftype=data_fty, prod_name=prod $ else message, pgm+': File <'+data_fnm+'> could not be read.' fnm = file_search(trk_fnm, /EXPAND_TILDE, /TEST_READ) if (fnm ne "" ) then $ out_track, iband=1, ifile=fnm, skip=0, ofile=out_fnm, /no_connect $ else message, pgm+': File <'+trk_fnm+'> could not be not read.'

By dash Date 2009-12-15 18:35

Dear Mike,

Now the program runs fine with the MODIS data. Thanks a lot.

My main aim to do this was to be able to extract pixel values from some of the OCM1 L2 images I processed. Bryan had told me that the OCM1 L2 file should be viewable with SeaDAS. I am able to display these L2 files in SeaDAS and also I am able to extract pixel values in the interactive mode. Can we extract pixel values from these L2 files in command mode too?

I changed the program to extract pixel values from one of these L2 files, and I am getting the following error:

seadas@seadasva:~/shared/Data/lda_hdf$ seadas -b lda_output1
IDL Version 7.0 (linux x86 m32). (c) 2007, ITT Visual Information Solutions

% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
% Embedded IDL: NASA GSFC SeaDAS Development, SeaDAS.
SeaDAS Version 6.0 (pid = 9565)
[LOAD] - Invalid file type for keyword FTYPE
SDP_VAL_BAND: No band available for band_no : 1
% Attempt to subscript FREEBANDFLAG (SDS_BAND) with BAND_IX is out of range.
[MAIN]: Error executing the command:
if (fnm ne "" ) then out_track, iband=1, ifile=fnm, skip=0, ofile=out_fnm, /no_connect else message, pgm+': File <'+trk_fnm+'> could not be not read.'
MAIN: Error executing command in batch file
seadas@seadasva:~/shared/Data/lda_hdf$

Thanks for your time,
Regards,
Dash
P.S. Please find the program, lat/lon file and the L2 file used attached. Could we add few lines in the program for projection of the file first (the SeaDAS default projection is fine), and then do the pixel value extraction?

Attachment: lda_output1 (0.5k)

Attachment: ldalatlon1.dat (0.3k)

Attachment: o01.06321.1801.r.L2.unmasked.hdf (411.8k)

By mike Date 2009-12-15 21:23

Just leave out the ftype argument in the load command, e.g.:

if (fnm ne "" ) then load, fnm(0), prod_name=prod $ else message, pgm+': File <'+data_fnm+'> could not be read.'

By dash Date 2009-12-17 01:21

Dear Mike,

Thank you so much. It worked. Now I can get the output for one variable. However, when I am trying to extract more than one variable, it runs fine but in the output text file I am getting pixel values of just one variable.

Further, I tried to map/project the variable first, and then extract the pixel values, but I am getting an error message saying,"[LOAD] - No valid product name selected".

Please find the updated program attached. If we could make this program to output multiple variables and preferably, if we could do the projection first and then extract the pixel values, it would be great.

Thank you very much for your time.
Regards,
Dash

Attachment: lda_output1 (1.1k)

By mike Date 2009-12-17 17:57

Dash, I'll simplify your script a bit so it will be more easy to understand, and then you're going to have to start learning how to write and troubleshoot programs on your own. If something isn't working you need to just keep simplifying the script until you can figure out the line that is causing the problem. You can then focus on that one line and play with it until it is working and then go on to the next line. If you plan to do a lot of IDL programming I'd advise you to buy it since then you can figure out bugs right on the IDL command-line instead of having to use the runtime SeaDAS batch option to run the entire script each time. So here's an example of something similar to what you asked for.. in this example the 'ldalatlon1.dat'' file contains the text: 29.92 -90.62 1

data_fnm='o01.06321.1801.r.L2.unmasked.hdf' trk_fnm='ldalatlon1.dat' out_fnm='321_output_test.txt' prod=['Rrs_414','Rrs_441','Rrs_486','Rrs_511','Rrs_556','Rrs_669'] load, data_fnm, prod_name=prod mapimg, bands=[1,2,3,4,5,6] for i = 1,6 do begin & $ out_track, iband=i+6, ifile= trk_fnm, skip=0, ofile=out_fnm, /no_connect, /append & $ endfor

This creates the '321_output_test.txt' text file with pixels extracted for each mapped band. You could then use this Unix command to clean up the extraneous text in this file:

grep -v Image 321_output_test.txt |grep -v Seq |grep -v "^ $" |grep -ve "---" >final_output.txt

By dash Date 2009-12-17 18:50

Dear Mike,

Thank you very much. Now I can extract all the variables. I really appreciate your help.

Actually, we have IDL on our linux machines but they have an old SeaDAS version on them. I am not conversant with installing software in linux, therefore I installed new SeaDAS VA easily on my windows laptop and working on it. As per your advice, I will run my programs on IDL after our sys. adm. installs the new SeaDAS on the linux machines, so that I can troubleshoot them line-by-line.

I am sorry for your time on this. You have a great day!
Regards,
Dash

By mike Date 2009-12-17 18:53

Ok glad it's working for you Dash, and you're on your way to becoming an IDL geek.