How to read zipped fies without open them

I have a bunch of zipped rain radar files which are raster. I cannot open all of them, becasue they are many and large. Is there a way to read them without opening them? I would like to read the by the date in the file name and sum them.

Thank you in advance

Answer 1

[Walter Roberson]

See https://www.mathworks.com/matlabcentral/answers/111836-how-to-unzip-files-using-external-unzip-software#answer_120392

and see also https://www.mathworks.com/matlabcentral/fileexchange/77257-zipfile?s_tid=srchtitle

[Shulamit Nussboim]

Thank you, but zipped file names ending are not zip but asc:

e.g.: 'RMcomp_201910150040.asc'

[Walter Roberson]

In that first link, in the code section "UPDATED VERSION FOR SCANNING A FOLDER FOR ZIP FILES AND PROCESS ALL OF THEM", change the .zip to .asc

[Shulamit Nussboim]

Thank you!

[Shulamit Nussboim]

I tried the code and I have two problems:

1. When I run the line

    zipFile = org.apache.tools.zip.ZipFile(fullfile(zipfolder,zipfiles(i).name));

I get this:

Java exception occurred:
java.util.zip.ZipException: archive is not a ZIP archive

	at org.apache.tools.zip.ZipFile.positionAtEndOfCentralDirectoryRecord(ZipFile.java:774)

	at org.apache.tools.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:710)

	at org.apache.tools.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:455)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:216)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:191)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:163)
    
    
    

2. I cannot run the code on a folder with a lagre number of files. The current folder says:" There are too many entries to display. Click the search botton in the adreess bar to reduce the number of entries".

[Walter Roberson]

That file extension tends to suggest that the file is not zip and is instead a ASCII Grid radar file.

https://www.mathworks.com/help/map/ref/arcgridread.html

[Shulamit Nussboim]

Yes, but it is zipped (ending is asc.gz). If I use 'readgeoraster' for the uzipped file it reads it, but I have a bunch of files and I cannot open each one but I want to run over the directory and read it in a loop. The names are (the gz ending does not appear in the folder but only in the current folder in the matlab):

RMcomp_201910150000.asc

RMcomp_201910150010.asc

RMcomp_201910150020.asc

RMcomp_201910150030.asc

etc.

I want to read/ extract data and sum them, and to leave all the files zipped after I extracted from them the data, because they are very large.

Thank you for you help

[Walter Roberson]

I am confused about whether the zip files do or do not end in .gz .

If the zip files end in .gz then In that first link, in the code section "UPDATED VERSION FOR SCANNING A FOLDER FOR ZIP FILES AND PROCESS ALL OF THEM", change the .zip to .gz

[Shulamit Nussboim]

Thank you, I will tried. It Shows me again:

Error using unzipascfile (line 7)
Java exception occurred:
java.util.zip.ZipException: archive is not a ZIP archive

	at org.apache.tools.zip.ZipFile.positionAtEndOfCentralDirectoryRecord(ZipFile.java:774)

	at org.apache.tools.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:710)

	at org.apache.tools.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:455)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:216)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:191)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:163)

 

The files are zipped and appear ending as .asc,gz. tried to change to gz and asc.gz and it did not work (changed on the following line, instead of zip):

    zipfiles = dir([zipfolder,'\*.zip']);

and it did not work.

Maybe my explenation is not well understood. I will try another approach and thank you for the attention and the time you dedicated.

[Walter Roberson]

The files are zipped and appear ending as .asc,gz

Is that a comma before the gz ? If so then

    zipfiles = dir([zipfolder,'\*,gz']);

Are you using Windows? Or are you using linux or Mac? If you are using Linux or Mac, then try

    !file RMcomp_201910150000.asc.gz

to see what the operating system can guess about what kind of file it is. (This will not work for Windows unless you install the Linux subsystem.)

[Shulamit Nussboim]

Sorry, it was a typing mistake. It is .gz and I use windows 10.

I tried

zipfiles = dir([zipfolder,'\*.gz']);

but still it did not work and starting the first iteration the error apeared again:

Error using unzipascfile (line 7)
Java exception occurred:
java.util.zip.ZipException: archive is not a ZIP archive

	at org.apache.tools.zip.ZipFile.positionAtEndOfCentralDirectoryRecord(ZipFile.java:774)

	at org.apache.tools.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:710)

	at org.apache.tools.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:455)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:216)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:191)

	at org.apache.tools.zip.ZipFile.<init>(ZipFile.java:163)

Maybe I have to look fore a different solution. Thank you

[Walter Roberson]

Well, you can use gunzip() to uncompress the file, but that expand the file on disk. There is no tool that can "stream" an unzipped gz file into arcgridread() . You would have to merge the code for gunzip and arcgridread(), which would take a fair bit of modification.

It would probably be easier to loop gunzip one file at a time, read it, delete the unzipped version .

[Shulamit Nussboim]

Yes, thank you. I find this direction usefull. Do you have any idea why the code did not work for me?

[Walter Roberson]

org.apache.tools.zip.ZipFile() works with .zip files but .gz files are a slightly different format. gunzip() knows how to read them.

[Shulamit Nussboim]

Thank you