Processing huge volume data file using powershell

Question

I am trying to perform a replace operation on a data file which is 4GB. But I am not even able to read this file due to memory exception. The following command gives a memory error.

$edwfile = (Get-Content C:\Users\tomgeorg\Desktop\edw_ord_extr_3x_SIQP_20181021.182305\edw_ord_extr_3x_SIQP_20181021.182305.dat -Raw )

Is there any alternative commands or tricks to process huge file.

I want to run the following replace pattern on each line in the file.basically I want to remove all the unwanted special characters.

-replace  "[$([char]0x00)-$([char]0x09)$([char]0x0B)-$([char]0x1F)$([char]0x7F)-$([char]0xFF)]","?"

system details

Don't use -Raw with large files. You'll need to process it in chunks using either the pipeline, -ReadCount, -Stream, or some combination of the above. — Maximilian Burszley
– Maximilian Burszley, Commented Oct 24, 2018 at 16:01
NB: At the moment you're just showing how to load the entire file into a single variable; so we can't give you much more help than @TheIncorrigible1's previous comment. If you need more assistance, please share info on what your process does / ideally sharing the related code, and we can advise how to better implement this suggestion. — JohnLBevan
– JohnLBevan, Commented Oct 24, 2018 at 16:03
Also don't put Get-Content in a subexpression/grouping expression or assign its output to a variable. Use the pipeline for processing the file(s) one line at a time. — Ansgar Wiechers
– Ansgar Wiechers, Commented Oct 24, 2018 at 16:03
Some people use Get-Content -ReadCount. Many people turn to System.IO.StreamReader for performance gains. stackoverflow.com/questions/6855814/… — lit
– lit, Commented Oct 24, 2018 at 16:18

Mike Twc · Accepted Answer · 2018-10-24 16:55:53Z

3

Below is the sample solution with streams. It reads file line by line and then add updated line to a new file.

$reader = [System.IO.StreamReader]"C:\temp\OriginalFile.txt"
$writer = [System.IO.StreamWriter]"C:\temp\UpdatedFile.txt"

while (!$reader.EndOfStream) {

$writer.WriteLine(($reader.ReadLine() -replace '\|', ";"))

}

$reader.Close()
$writer.Close()

edited Oct 24, 2018 at 16:55

answered Oct 24, 2018 at 16:50

Mike Twc

2,3752 gold badges17 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Maximilian Burszley · Accepted Answer · 2018-10-24 16:06:20Z

2

Assuming you are expecting to work on one line at a time, you'll want to use the pipeline for your task:

$path = '~\Desktop\edw_ord_extr_3x_SIQP_20181021.182305\edw_ord_extr_3x_SIQP_20181021.182305.dat'
Get-Content -Path $path | ForEach-Object {
    # do something line-by-line with the file
} | # -> do something else with the output

Without knowing what you're doing with the file, it's hard to give a more complete answer.

answered Oct 24, 2018 at 16:06

Maximilian Burszley

20k7 gold badges39 silver badges68 bronze badges

Collectives™ on Stack Overflow

Processing huge volume data file using powershell

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related