How to iterate a text file for a specific key and save in variable using powershell?

Question

I am trying to create a script which will loop through a text file <-- (Step 3.1) and find for specific keys and saving the values to a variable. Now that variable I am using in a command to generate the output file < --(step 4). I am using the code as of now.

Code:

# 1- Connect to Azure Account

$username = "[email protected]"
$pass = ConvertTo-SecureString "abc123" -AsPlainText -Force
$cred = New-Object PSCredential($username,$pass)

#Connect-AzureRmAccount -Credential $cred | out-null
Connect-AzAccount -Credential $cred | out-null

# 2 - Input Area

$subscriptionName = 'Data Analytics'
$resourceGroupName = 'DataLake-Gen2'
 $dataFactoryName = 'dna-production-gen2'
$runStartDateTimeUTC = '2020-09-12T06:40:00Z'

# 3 - (All Triggers Information)

$ErrorActionPreference="SilentlyContinue"
Stop-Transcript | out-null
$ErrorActionPreference = "Continue"
Start-Transcript -path D:\Powershell\TriggerInfo.txt -append
Get-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName
Stop-Transcript  

# 3.1 (Get the content in a variable)

Get-Content "D:\Powershell\TriggerInfo.txt" | ForEach-Object {
$trg_name = $_.TriggerName
} $trg_name

# 4 - (Write the output to a text file, The Get-AzDataFactoryV2TriggerRun command returns detailed information about trigger runs for the specified trigger 
#  in the given timeframe.)

foreach ($trg in $trg_name) {

$ErrorActionPreference="SilentlyContinue"
Stop-Transcript | out-null
$ErrorActionPreference = "Continue"
Start-Transcript -path D:\Powershell\output.txt -append
Get-AzDataFactoryV2TriggerRun -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -TriggerName $trg -TriggerRunStartedAfter "2020-09-01" -TriggerRunStartedBefore "2020-09-17"
Stop-Transcript

"$trg = " + $trg.length
}

TriggerInfo.txt:

TriggerName       : TRG_CM_TBLEnhanced_prod
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_tblEnhanced_QA
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CustCaseData
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Dashboard_TAE
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

Any help will be appreciated? Thanks

Updated Code

    # 3.1 (Get the content in a variable)
    
    # Get-Content "D:\Powershell\TriggerInfo.txt" | ForEach-Object {
    # $trg_name = $_.TriggerName
    # } $trg_name
    
    # $trg = (Get-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName).TriggerName
    
    $log = Get-Content -Path 'D:\Powershell\TriggerInfo.txt' -Raw
    $result = ($log -split '(\r?\n){2,}' | Where-Object {$_ -match '\S'}) | ForEach-Object {
    [PsCustomObject](($_ -split 'TriggeredPipelines')[0] -replace ' : ', '=' | ConvertFrom-StringData)  |
        Select-Object 'TriggerName'
    }
    
    # output on screen
    $result | Format-Table -AutoSize
    
    # write to CSV file
    $result | Export-Csv -Path 'D:\Powershell\TriggerInforesult.csv' -Encoding UTF8 -NoTypeInformation -Force
    
    $trg_name = (Import-Csv -Path 'D:\Powershell\TriggerInforesult.csv').TriggerName
    
    # 4 - (Write the output to a text file, The Get-AzDataFactoryV2TriggerRun command returns detailed information about trigger runs for the specified trigger 
    #  in the given timeframe.)
    
    foreach ($trigger in $trg_name) {
    
    $ErrorActionPreference="SilentlyContinue"
    Stop-Transcript | out-null
    $ErrorActionPreference = "Continue"
    Start-Transcript -path D:\Powershell\output.txt -append
    
    Get-AzDataFactoryV2TriggerRun -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -TriggerName $trigger -TriggerRunStartedAfter "2020-09-01" -TriggerRunStartedBefore "2020-09-17"
    
    Stop-Transcript
    
    "$trigger = " + $trigger.length
    }

****Recent Update****

TriggerName       : TRG_CM_TBLEnhanced_prod
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_tblEnhanced_QA
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CustCaseData
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Dashboard_TAE
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : GMB_Trigger
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.BlobEventsTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Optimizely_Import
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_USBusinessData_Monthly
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_Generic_CSV_To_DW
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Stopped

TriggerName       : TRG_CM_Dimension_Unit
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_PricingQuoteApproval
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

Transcript stopped, output file is D:\Powershell\new\TriggerInfo.txt
ConvertFrom-StringData : Data line '**********************' is not in 'name=value' format.
At line:36 char:37
+     $data  = $_ -replace ':', '=' | ConvertFrom-StringData
+                                     ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [ConvertFrom-StringData], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.ConvertFromStringDataCommand

Get-AzDataFactoryV2TriggerRun : Cannot validate argument on parameter 'DataFactoryName'. The argument is null or empty. Provide an argument that is not null or empty, and then try the
command again.
At line:47 char:34
+    Get-AzDataFactoryV2TriggerRun @splat
+                                  ~~~~~~
    + CategoryInfo          : InvalidData: (:) [Get-AzDataFactoryV2TriggerRun], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.Azure.Commands.DataFactoryV2.GetAzureDataFactoryTriggerRunCommand

ConvertFrom-StringData : Data line '**********************' is not in 'name=value' format.
At line:36 char:37
+     $data  = $_ -replace ':', '=' | ConvertFrom-StringData
+                                     ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [ConvertFrom-StringData], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.ConvertFromStringDataCommand

TriggerInfo.txt

Windows PowerShell transcript start Start time: 20201017093947 Username: XXXXXX\XXXXXX RunAs User: XXXXXX\XXXXXX Configuration Name: Machine: INNOPHLTXETR138 (Microsoft Windows NT 10.0.17134.0) Host Application: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Process ID: 20340 PSVersion: 5.1.17134.858 PSEdition: Desktop PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.17134.858 BuildVersion: 10.0.17134.858 CLRVersion: 4.0.30319.42000 WSManStackVersion: 3.0 PSRemotingProtocolVersion: 2.3 SerializationVersion: 1.1.0.1

Transcript started, output file is D:\Powershell\new\TriggerInfo.txt

TriggerName       : TRG_CM_TBLEnhanced_prod
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_tblEnhanced_QA
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CustCaseData
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Dashboard_TAE
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : GMB_Trigger
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.BlobEventsTrigger
RuntimeState      : Started

TriggerName       : TRG_RP_Optimizely_Import
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_USBusinessData_Monthly
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_Generic_CSV_To_DW
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Stopped

TriggerName       : TRG_CM_Dimension_Unit
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

TriggerName       : TRG_CM_PricingQuoteApproval
ResourceGroupName : DataLake-Gen2
DataFactoryName   : dna-production-gen2
Properties        : Microsoft.Azure.Management.DataFactory.Models.ScheduleTrigger
RuntimeState      : Started

**********************
Windows PowerShell transcript end
End time: 20201017093951
**********************

You really making it yourself very difficult, this is not how PowerShell is supposed to work. You might simply grap your triggers from the Az.DataFactory object like this: (Get-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName).TriggerName — iRon
– iRon, Commented Oct 16, 2020 at 8:31
Hi iRon, Thanks for the suggestion. I am actually newbie to Powershell. Triggers info. I have already saved in the triggerinfo file, now I need my output file generated for each individual triggername and append the o/p to output file. — Saurabh
– Saurabh, Commented Oct 16, 2020 at 8:51
hmm, this really not the way to go. Anyways, this ConvertFrom-SourceTable cmdlet might help you (see also: Parsing PowerShell space-separated output as a table. If you add (a part) of the TriggerInfo.txt output (to the question), I might be able to be more specific. — iRon
– iRon, Commented Oct 16, 2020 at 9:06
As for your Step 3.1. Get-Content reads a file into a string array. From that you would need to parse it out in various properties. However, from your earlier question, you have all you need already in a structured CSV file, so why don't you use that like this: $trg_name = (Import-Csv -Path 'TheResultsFileFromEarlierQuestion.CSV').TriggerName. Then $trg_name will contain the array of TriggerNames you want. — Theo
– Theo, Commented Oct 16, 2020 at 9:54

Theo · Accepted Answer · 2020-10-17 14:08:36Z

0

As commented, Get-Content reads a file as string array. If you want to use the example textfile you gave in the question, you need parse it out as an array of objects in order to select the property or properties you want filtered out.

However, in your previous question, you have already done the hard work and have a structured CSV file with all the properties you'll ever want, so I suggest you use that for input.

The code can then be as simple as:

$trg_name = (Import-Csv -Path 'TheResultsFileFromEarlierQuestion.CSV').TriggerName

and use that to save the $trigger array to a file, or use it for iterating stuff:

$trg_name | ForEach-Object {
    # using splatting here for prettier code
    $splat = @{
        ResourceGroupName       = $resourceGroupName
        DataFactoryName         = $dataFactoryName
        TriggerName             = $_
        TriggerRunStartedAfter  = "2020-09-01"
        TriggerRunStartedBefore = "2020-09-17"
    }
    Get-AzDataFactoryV2TriggerRun @splat
} | Export-Csv -Path 'PathToTheOutputFile.csv' -Encoding UTF8 -NoTypeInformation

P.S. If $resourceGroupName and $dataFactoryName should also come from the input CSV file, then you don't need $trg_name at all and can simply do:

Import-Csv -Path 'TheResultsFileFromEarlierQuestion.CSV' | ForEach-Object {
    $splat = @{
        ResourceGroupName       = $_.ResourceGroupName
        DataFactoryName         = $_.DataFactoryName
        TriggerName             = $_.TriggerName
        TriggerRunStartedAfter  = "2020-09-01"
        TriggerRunStartedBefore = "2020-09-17"
    }
    Get-AzDataFactoryV2TriggerRun @splat
} | Export-Csv -Path 'PathToTheOutputFile.csv' -Encoding UTF8 -NoTypeInformation

and that then would be the complete code for both Step 3.1 and Step 4

Edit

From your comments, I understand that you can not use the csv file you made earlier for this, and that you're stuck with the input textfile as you show us.

That too is not very hard to use:

# split the text in 'trigger' text blocks on the empty line
$triggers = ((Get-Content "D:\Powershell\TriggerInfo.txt" -Raw) -split '\*+')[0] -split '(\r?\n){2,}'
# loop through these blocks (skip any possible empty textblock)
$triggers | Where-Object {$_ -match '\S'} | ForEach-Object {
    # and parse the data into Hashtables
    $data  = $_ -replace ':', '=' | ConvertFrom-StringData
    # if you have values containing a colon (:) character, this is safer:
    # $data  = ($_ -split ':', 2) -join '=' | ConvertFrom-StringData

    $splat = @{ 
        ResourceGroupName       = $data.ResourceGroupName
        DataFactoryName         = $data.DataFactoryName
        TriggerName             = $data.TriggerName
        TriggerRunStartedAfter  = "2020-10-15"
        TriggerRunStartedBefore = "2020-10-17"
   } 
   Get-AzDataFactoryV2TriggerRun @splat 
} | Export-Csv -Path 'D:\Powershell\new\Output.csv' -Encoding UTF8 -NoTypeInformation

edited Oct 17, 2020 at 14:08

answered Oct 16, 2020 at 10:33

Theo

61.9k8 gold badges28 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Saurabh Over a year ago

Thanks, I'll implement the same. Your help is appreciated.

Saurabh Over a year ago

Hi Theo, This is the error I am getting "Get-AzDataFactoryV2TriggerRun : Cannot validate argument on parameter 'Name'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again." I can't use the file output from previous question, because that output was only one trigger name, I want to generate the output for each triggername and append it in the output file. That is why I am using the triggerinfo file to store and loop through the name of the trigger and pass it as a parameter to AzDataFactoryV2TriggerRun. Hope you get it.

Saurabh Over a year ago

$trg_name = (Import-Csv -Path 'D:\Powershell\new\TriggerInfo.csv').TriggerName $trg_name | ForEach-Object { # using splatting here for prettier code $splat = @{ ResourceGroupName = $resourceGroupName DataFactoryName = $dataFactoryName TriggerName = $_ TriggerRunStartedAfter = "2020-10-15" TriggerRunStartedBefore = "2020-10-17" } Get-AzDataFactoryV2TriggerRun @splat } | Export-Csv -Path 'D:\Powershell\new\Output.csv' -Encoding UTF8 -NoTypeInformation

Saurabh Over a year ago

This is what I am using.

Theo Over a year ago

@SaurabhShakyawar Please see the last edit. The code now uses the textfile and converts that to usable data to use.

|

Collectives™ on Stack Overflow

How to iterate a text file for a specific key and save in variable using powershell?

1 Answer 1

11 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related