4

This is my first post - I will be happy to make any corrections required for any mistakes made in the post.

I have been looking through the forums here for a few months and have learned a lot but I cannot seem to accomplish my goal with what I have found.

I need to read a CSV file (Read-Only) when it changes and place the resulting array into and active and open Excel 2016 Tab. I can do this using com and system.io.watcherchangetypes but this is too slow and requires copy paste.

I need to read the csv as fast as possible (under a second) and convert the lines into a usable array for Excel. This whole process has to take under 2 seconds MAX. Some of the CSV's will exceed 180,000 lines as the day goes on.

I work for a Trading Company.

I would be happy with a single column, Tab delimited and multiple Rows. I cant get the multiple rows.

I have to write the range line by line and that takes too long.

I was looking at this one but I am not clear on how to make the whole thing dynamic. There is no set amount of headers and the rows will change as well. I cannot work with any static data at all.

This is the post which prompted me to ask for help: How to use powershell to reorder CSV columns

$export = "\\UNC\to\file\Name.csv"
#$excel = New-Object -ComObject Excel.Application
#$excel.visible = $true
#$workbook = $excel.Workbooks.Add()

$reader = [System.IO.File]::OpenText($export)
$writer = New-Object System.IO.StreamWriter "data2.csv"
for(;;) {
    $line = $reader.ReadLine()
    if ($null -eq $line) {
        break
    }
    $i=1
    $data = $line.Split(",") | %{
    if($_ -ne $null)
    {
        Write-Host $_ $i
        ++$i
    }
    }
    [void]$data.Length
   # $data.GetValue()
    #$writer.WriteLine('{0},{1},{2}', $data[0], $data[1], $data[2])
}
$reader.Close()
#$writer.Close()

Any help will be greatly appreciated!

UPDATE:

I figured it out. The result is probably not the most efficient but it gets me what I need for now while i explore how to better accomplish it with what I have learned.

(Measure-Command { $data = [System.io.File]::Open($export, 'Open', 'Read', 'ReadWrite')
$reader = New-Object System.IO.StreamReader($data)
$count = 0
While($text = $reader.Readline())
{
    If($text -eq $null)
    {
        $reader.Close()
        $data.close()
    }
    ++$count
}
}).TotalSeconds

$array2 = New-Object 'object[,]' $count,1
$end = ++$count
$file = New-Object System.IO.StreamReader -ArgumentList $export
$stringBuilder = New-Object System.Text.StringBuilder
$list = New-Object System.Collections.Generic.List[System.String]
$a = 0
Measure-Command {
    While ($i = $file.ReadLine() -Replace ",","`t")
    {
        if ($i -eq $null)
        {
            $file.close()
            break loop
        }
        $null = $stringBuilder.Append($i)
        $list.Add($i)
        $array2[$a,0] = $i
        ++$a
    }

    $outputString = $stringBuilder.ToString()
    $array = $list.ToArray()
}
7
  • 2
    Personally, I'd use Excel to open the CSV as a data file...see superuser.com/questions/533018/… Commented Sep 17, 2018 at 23:39
  • 1
    Does it need to be an existing, open Excel tab? Using the COM object is inherently slow. Commented Sep 17, 2018 at 23:48
  • Opening it as a data file is used by some of them but a one minute refresh is too long on heavy volume days. They need to import the data almost on demand. Commented Sep 18, 2018 at 11:21
  • No it does not need to be an existing tab but this is how they want it. Commented Sep 18, 2018 at 11:22
  • What was the problem making you not using Import-CSV? Commented Sep 18, 2018 at 11:55

1 Answer 1

0

You can do something like this

data = pd.read_csv("data1.csv", sep='\s+',header=None)
dataarraynew13phse = np.array(data)
dataarraynew13phse=dataarraynew13phse.flatten()

sep = '\s+' can be useful to decode tabs, in multiple lines And then flatten() can make it in a single row or array

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.