Friday, June 17, 2011

PowerShell Tail Improvements

Here are the latest improvements I have made to my tail functions for PowerShell. I typically work in a heterogenous computing environment so I need to be able to handle the various text file encodings and new line delimiters I encounter. The previous blogs posts (here, here & here) showed a steady improvement and functionality with the major feature I needed to complete being the detecting and reading of text files that were not were ASCII encoded with Windows end-of-line (CR+LF). This stops hard coding specific changes in my code to deal with each situation I encounter.

I have added two functions to the code to handle these two items. Looking at the head of the file, I attempt to detect the byte order mark (BOM) to determine if the file is unicode encoded and its endianess. If I am unable to make that determination, revert to ASCII as the encoding. I work with the System.Text.Encoding class to assist in the decode of the unicode based text files. The second function detects the new line delimiter by searching the head for the match of Windows (CR+LF), UNIX (LF) or Classic Macintosh (CR) to assist in the breaking of the lines for initial tail read.

In the code sample below, you will find these two new functions with the log file "tailed" being a system.log from a hypothetical Mac OS X server sharing out its "var" directory via SAMBA so we can access the system.log file in the log subdirectory. This file is typically ASCII encoded with UNIX new lines.

If you are running Microsoft Forefront Client Security and want to monitor the updates, virus detections and removals, you need to access the "MPLog-*.log" file which is Unicode-16 Little-Endian encoded. Swap out the inputFile variable to watch this file for updates. You can find that file here:
$env:ALLUSERSPROFILE\Application Data\Microsoft\Microsoft Forefront\Client Security\Client\Antimalware\Support
These are good examples to demonstrate the capability of the added functions add flexibility to my prior attempts.
Function Get-FileEncoding($fileStream) {
 $fileEncoding = $null
 if($fileStream.CanSeek) {
  [byte[]] $bytesToRead = New-Object byte[] 4
  $fileStream.Read($bytesToRead, 0, 4) | Out-Null
  if($bytesToRead[0] -eq 0x2B -and  $bytesToRead[1] -eq 0x2F -and  $bytesToRead[2] -eq 0x76) { # UTF-7
   $encoding = "utf7"
  } elseif($bytesToRead[0] -eq 0xFF -and $bytesToRead[1] -eq 0xFE) { # UTF-16 Little-Endian
   $encoding = "unicode-le"
  } elseif($bytesToRead[0] -eq 0xFE -and $bytesToRead[1] -eq 0xFF) { # UTF-16 Big-Endian
   $encoding = "unicode-be"
  } elseif($bytesToRead[0] -eq 0 -and $bytesToRead[1] -eq 0 -and $bytesToRead[2] -eq 0xFE -and $bytesToRead[3] -eq 0xFF) { # UTF-32 Big Endian
   $encoding = "utf32-be"
  } elseif($bytesToRead[0] -eq 0xFF -and $bytesToRead[1] -eq 0xFE -and $bytesToRead[2] -eq 0 -and $bytesToRead[3] -eq 0) { # UTF-32 Little Endian
   $encoding = "utf32-le"
  } elseif($bytesToRead[0] -eq 0xDD -and $bytesToRead[1] -eq 0x73 -and $bytesToRead[2] -eq 0x66 -and $bytesToRead[3] -eq 0x73) { # UTF-EBCDIC
   $encoding = "unicode"
  } elseif($bytesToRead[0] -eq 0xEF -and $bytesToRead[1] -eq 0xBB -and $bytesToRead[2] -eq 0xBF) { # UTF-8 with BOM
   $encoding = "utf8"
  } else { # ASCII Catch-All
   $encoding = "ascii"
  }
  switch($encoding) {
   "unicode-be" { $fileEncoding = New-Object System.Text.UnicodeEncoding($true, $true) }
   "unicode-le" { $fileEncoding = New-Object System.Text.UnicodeEncoding($false, $true) }
   "utf32-be" { $fileEncoding = New-Object System.Text.UTF32Encoding($true, $true) }
   "utf32-le" { $fileEncoding = New-Object System.Text.UTF32Encoding($false, $true) }
   "unicode" { $fileEncoding = New-Object System.Text.UnicodeEncoding($true, $true) }
   "utf7" { $fileEncoding = New-Object System.Text.UTF7Encoding } 
   "utf8" { $fileEncoding = New-Object System.Text.UTF8Encoding } 
   "utf32" { $fileEncoding = New-Object System.Text.UTF32Encoding } 
   "ascii" { $fileEncoding = New-Object System.Text.AsciiEncoding }
  }
 }
 return $fileEncoding 
}
#--------------------------------------------------------------------------------------------------#
Function Get-NewLine($fileStream, $fileEncoding) {
 $newLine = $null
 $byteChunk = 512
 if($fileStream.CanSeek) {
  $fileSize = $fileStream.Length
  if($fileSize -lt $byteChunk) { $byteChunk -eq $fileSize }
  [byte[]] $bytesToRead = New-Object byte[] $byteChunk
  $fileStream.Read($bytesToRead, 0, $byteChunk) | Out-Null
  $testLines = $fileEncoding.GetString($bytesToRead)
  if($testLines -match "\r\n") { # Windows
   $newLine = "\r\n"
  } elseif($testLines -match "\n") { # Unix
   $newLine = "\n"
  } elseif($testLines -match "\r") { # Classic Mac
   $newLine = "\r"
  } else { # When all else fails, Go Windows
   $newLine = "\r\n"
  }
 }
 return $newLine
}
#--------------------------------------------------------------------------------------------------#
Function Read-EndOfFileByByteChunk($fileName,$totalNumberOfLines,$byteChunk) {
 if($totalNumberOfLines -lt 1) { $totalNumberOfLines = 1 }
 if($byteChunk -le 0) { $byteChunk = 10240 }
 $linesOfText = New-Object System.Collections.ArrayList
 if([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $fileEncoding = Get-FileEncoding $fileStream
  $newLine = Get-NewLine $fileStream $fileEncoding
  $fileSize = $fileStream.Length
  $byteOffset = $byteChunk
  [byte[]] $bytesRead = New-Object byte[] $byteChunk
  $totalBytesProcessed = 0
  $lastReadAttempt = $false
  do {
   if($byteOffset -ge $fileSize) {
    $byteChunk = $fileSize - $totalBytesProcessed
    [byte[]] $bytesRead = New-Object byte[] $byteChunk
    $byteOffset = $fileSize
    $lastReadAttempt = $true
   }
   $fileStream.Seek((-$byteOffset), [System.IO.SeekOrigin]::End) | Out-Null
   $fileStream.Read($bytesRead, 0, $byteChunk) | Out-Null
   $chunkOfText = New-Object System.Collections.ArrayList
   $chunkOfText.AddRange(([System.Text.RegularExpressions.Regex]::Split($fileEncoding.GetString($bytesRead),$newLine)))
   $firstLineLength = ($chunkOfText[0].Length)
   $byteOffset = ($byteOffset + $byteChunk) - ($firstLineLength)
   if($lastReadAttempt -eq $false -and $chunkOfText.count -lt $totalNumberOfLines) {
    $chunkOfText.RemoveAt(0)
   }
   $totalBytesProcessed += ($byteChunk - $firstLineLength)
   $linesOfText.InsertRange(0, $chunkOfText)
  } while($totalNumberOfLines -ge $linesOfText.count -and $lastReadAttempt -eq $false -and $totalBytesProcessed -lt $fileSize)
  $fileStream.Close()
  if($linesOfText.count -gt 1) {
   $linesOfText.RemoveAt($linesOfText.count-1)
  }
  $deltaLines = ($linesOfText.count - $totalNumberOfLines)
  if($deltaLines -gt 0) {
   $linesOfText.RemoveRange(0, $deltaLines)
  }
 } else {
  $linesOfText.Add("[ERROR] $fileName not found") | Out-Null
 }
 Write-Host $linesOfText.count
 return $linesOfText
}
#--------------------------------------------------------------------------------------------------#
Function Read-FileUpdates($fileName,$startSize) {
 if([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $fileEncoding = Get-FileEncoding $fileStream
  $fileStream.Close()
  while([System.IO.File]::Exists($fileName)) {
   $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
   if($fileStream.CanSeek) {
    $currentFileSize = $fileStream.Length
    if($currentFileSize -gt $startSize) {
     $byteChunk = $currentFileSize - $startSize
     [byte[]] $bytesRead = New-Object byte[] $byteChunk
     $fileStream.Seek((-$byteChunk), [System.IO.SeekOrigin]::End) | Out-Null
     $fileStream.Read($bytesRead, 0, $byteChunk) | Out-Null
     Write-Host ($fileEncoding.GetString($bytesRead)) -noNewLine
     $startSize = $currentFileSize
     }
    }
   $fileStream.Close()
   Start-Sleep -milliseconds 250
  }
 }
}
#--------------------------------------------------------------------------------------------------#
Set-Variable -name inputFile -option Constant -value "\\macosx-server.mydomain.local\var\log\system.log"
#--------------------------------------------------------------------------------------------------#
if([System.IO.File]::Exists($inputFile)) {
 Write-Host (Read-EndOfFileByByteChunk $inputFile 10 1280 | Out-String) -noNewLine
 $fileStream = New-Object System.IO.FileStream($inputFile,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
 $fileSize = $fileStream.Length
 $fileStream.Close()
 Read-FileUpdates $inputFile $fileSize
} else {
 Write-Host "Could not find $inputFile..." -foregroundColor Red
}

No comments:

Post a Comment