Clone NTFS Permissions Fast with PowerShell

Need to clone NFTS permissions between two folders or files? The Get-Acl & Set-Acl commandlets provide this capability when used in conjunction with each other. With the one-liner below, you can transfer complex permissions from source to destination in seconds.

Set-Acl -path "\\fileserver.asia.ad.local\dept\target_directory" -aclObject (Get-Acl -path "\\fileserver.europe.ad.local\dept\source_directory")

PowerShell Tail Improvements

Here are the latest improvements I have made to my tail functions for PowerShell. I typically work in a heterogenous computing environment so I need to be able to handle the various text file encodings and new line delimiters I encounter. The previous blogs posts (here, here & here) showed a steady improvement and functionality with the major feature I needed to complete being the detecting and reading of text files that were not were ASCII encoded with Windows end-of-line (CR+LF). This stops hard coding specific changes in my code to deal with each situation I encounter.

I have added two functions to the code to handle these two items. Looking at the head of the file, I attempt to detect the byte order mark (BOM) to determine if the file is unicode encoded and its endianess. If I am unable to make that determination, revert to ASCII as the encoding. I work with the System.Text.Encoding class to assist in the decode of the unicode based text files. The second function detects the new line delimiter by searching the head for the match of Windows (CR+LF), UNIX (LF) or Classic Macintosh (CR) to assist in the breaking of the lines for initial tail read.

In the code sample below, you will find these two new functions with the log file "tailed" being a system.log from a hypothetical Mac OS X server sharing out its "var" directory via SAMBA so we can access the system.log file in the log subdirectory. This file is typically ASCII encoded with UNIX new lines.

If you are running Microsoft Forefront Client Security and want to monitor the updates, virus detections and removals, you need to access the "MPLog-*.log" file which is Unicode-16 Little-Endian encoded. Swap out the inputFile variable to watch this file for updates. You can find that file here:

$env:ALLUSERSPROFILE\Application Data\Microsoft\Microsoft Forefront\Client Security\Client\Antimalware\Support

These are good examples to demonstrate the capability of the added functions add flexibility to my prior attempts.

Function Get-FileEncoding($fileStream) {
 $fileEncoding = $null
 if($fileStream.CanSeek) {
  [byte[]] $bytesToRead = New-Object byte[] 4
  $fileStream.Read($bytesToRead, 0, 4) | Out-Null
  if($bytesToRead[0] -eq 0x2B -and  $bytesToRead[1] -eq 0x2F -and  $bytesToRead[2] -eq 0x76) { # UTF-7
   $encoding = "utf7"
  } elseif($bytesToRead[0] -eq 0xFF -and $bytesToRead[1] -eq 0xFE) { # UTF-16 Little-Endian
   $encoding = "unicode-le"
  } elseif($bytesToRead[0] -eq 0xFE -and $bytesToRead[1] -eq 0xFF) { # UTF-16 Big-Endian
   $encoding = "unicode-be"
  } elseif($bytesToRead[0] -eq 0 -and $bytesToRead[1] -eq 0 -and $bytesToRead[2] -eq 0xFE -and $bytesToRead[3] -eq 0xFF) { # UTF-32 Big Endian
   $encoding = "utf32-be"
  } elseif($bytesToRead[0] -eq 0xFF -and $bytesToRead[1] -eq 0xFE -and $bytesToRead[2] -eq 0 -and $bytesToRead[3] -eq 0) { # UTF-32 Little Endian
   $encoding = "utf32-le"
  } elseif($bytesToRead[0] -eq 0xDD -and $bytesToRead[1] -eq 0x73 -and $bytesToRead[2] -eq 0x66 -and $bytesToRead[3] -eq 0x73) { # UTF-EBCDIC
   $encoding = "unicode"
  } elseif($bytesToRead[0] -eq 0xEF -and $bytesToRead[1] -eq 0xBB -and $bytesToRead[2] -eq 0xBF) { # UTF-8 with BOM
   $encoding = "utf8"
  } else { # ASCII Catch-All
   $encoding = "ascii"
  }
  switch($encoding) {
   "unicode-be" { $fileEncoding = New-Object System.Text.UnicodeEncoding($true, $true) }
   "unicode-le" { $fileEncoding = New-Object System.Text.UnicodeEncoding($false, $true) }
   "utf32-be" { $fileEncoding = New-Object System.Text.UTF32Encoding($true, $true) }
   "utf32-le" { $fileEncoding = New-Object System.Text.UTF32Encoding($false, $true) }
   "unicode" { $fileEncoding = New-Object System.Text.UnicodeEncoding($true, $true) }
   "utf7" { $fileEncoding = New-Object System.Text.UTF7Encoding } 
   "utf8" { $fileEncoding = New-Object System.Text.UTF8Encoding } 
   "utf32" { $fileEncoding = New-Object System.Text.UTF32Encoding } 
   "ascii" { $fileEncoding = New-Object System.Text.AsciiEncoding }
  }
 }
 return $fileEncoding 
}
#--------------------------------------------------------------------------------------------------#
Function Get-NewLine($fileStream, $fileEncoding) {
 $newLine = $null
 $byteChunk = 512
 if($fileStream.CanSeek) {
  $fileSize = $fileStream.Length
  if($fileSize -lt $byteChunk) { $byteChunk -eq $fileSize }
  [byte[]] $bytesToRead = New-Object byte[] $byteChunk
  $fileStream.Read($bytesToRead, 0, $byteChunk) | Out-Null
  $testLines = $fileEncoding.GetString($bytesToRead)
  if($testLines -match "\r\n") { # Windows
   $newLine = "\r\n"
  } elseif($testLines -match "\n") { # Unix
   $newLine = "\n"
  } elseif($testLines -match "\r") { # Classic Mac
   $newLine = "\r"
  } else { # When all else fails, Go Windows
   $newLine = "\r\n"
  }
 }
 return $newLine
}
#--------------------------------------------------------------------------------------------------#
Function Read-EndOfFileByByteChunk($fileName,$totalNumberOfLines,$byteChunk) {
 if($totalNumberOfLines -lt 1) { $totalNumberOfLines = 1 }
 if($byteChunk -le 0) { $byteChunk = 10240 }
 $linesOfText = New-Object System.Collections.ArrayList
 if([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $fileEncoding = Get-FileEncoding $fileStream
  $newLine = Get-NewLine $fileStream $fileEncoding
  $fileSize = $fileStream.Length
  $byteOffset = $byteChunk
  [byte[]] $bytesRead = New-Object byte[] $byteChunk
  $totalBytesProcessed = 0
  $lastReadAttempt = $false
  do {
   if($byteOffset -ge $fileSize) {
    $byteChunk = $fileSize - $totalBytesProcessed
    [byte[]] $bytesRead = New-Object byte[] $byteChunk
    $byteOffset = $fileSize
    $lastReadAttempt = $true
   }
   $fileStream.Seek((-$byteOffset), [System.IO.SeekOrigin]::End) | Out-Null
   $fileStream.Read($bytesRead, 0, $byteChunk) | Out-Null
   $chunkOfText = New-Object System.Collections.ArrayList
   $chunkOfText.AddRange(([System.Text.RegularExpressions.Regex]::Split($fileEncoding.GetString($bytesRead),$newLine)))
   $firstLineLength = ($chunkOfText[0].Length)
   $byteOffset = ($byteOffset + $byteChunk) - ($firstLineLength)
   if($lastReadAttempt -eq $false -and $chunkOfText.count -lt $totalNumberOfLines) {
    $chunkOfText.RemoveAt(0)
   }
   $totalBytesProcessed += ($byteChunk - $firstLineLength)
   $linesOfText.InsertRange(0, $chunkOfText)
  } while($totalNumberOfLines -ge $linesOfText.count -and $lastReadAttempt -eq $false -and $totalBytesProcessed -lt $fileSize)
  $fileStream.Close()
  if($linesOfText.count -gt 1) {
   $linesOfText.RemoveAt($linesOfText.count-1)
  }
  $deltaLines = ($linesOfText.count - $totalNumberOfLines)
  if($deltaLines -gt 0) {
   $linesOfText.RemoveRange(0, $deltaLines)
  }
 } else {
  $linesOfText.Add("[ERROR] $fileName not found") | Out-Null
 }
 Write-Host $linesOfText.count
 return $linesOfText
}
#--------------------------------------------------------------------------------------------------#
Function Read-FileUpdates($fileName,$startSize) {
 if([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $fileEncoding = Get-FileEncoding $fileStream
  $fileStream.Close()
  while([System.IO.File]::Exists($fileName)) {
   $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
   if($fileStream.CanSeek) {
    $currentFileSize = $fileStream.Length
    if($currentFileSize -gt $startSize) {
     $byteChunk = $currentFileSize - $startSize
     [byte[]] $bytesRead = New-Object byte[] $byteChunk
     $fileStream.Seek((-$byteChunk), [System.IO.SeekOrigin]::End) | Out-Null
     $fileStream.Read($bytesRead, 0, $byteChunk) | Out-Null
     Write-Host ($fileEncoding.GetString($bytesRead)) -noNewLine
     $startSize = $currentFileSize
     }
    }
   $fileStream.Close()
   Start-Sleep -milliseconds 250
  }
 }
}
#--------------------------------------------------------------------------------------------------#
Set-Variable -name inputFile -option Constant -value "\\macosx-server.mydomain.local\var\log\system.log"
#--------------------------------------------------------------------------------------------------#
if([System.IO.File]::Exists($inputFile)) {
 Write-Host (Read-EndOfFileByByteChunk $inputFile 10 1280 | Out-String) -noNewLine
 $fileStream = New-Object System.IO.FileStream($inputFile,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
 $fileSize = $fileStream.Length
 $fileStream.Close()
 Read-FileUpdates $inputFile $fileSize
} else {
 Write-Host "Could not find $inputFile..." -foregroundColor Red
}

Replicating UNIX "tail -f" in PowerShell

Building upon my previous blog post, Unix Tail-like Functionality in PowerShell Revisited, I have completed the next step in providing tail functionality in my PowerShell scripts; the ability to emulate the "-f" argument.

From the TAIL(1) man page:

The -f option causes tail to not stop when end of file is reached, but rather to wait for additional data to be appended to the input.

My approach to replicating this functionality in PowerShell is to once again take advantage of System.IO.FileStream Class and read from the end of file as I did in my previous tail emulations. Originally, I thought this was going to be a more difficult task to accomplish but since I learned so much from my prior attempts, it turned out to be fairly simple and a lot less code to implement. My solution focuses on the fact that the file size grows as more data is appended to the text file. If I know how large the file was in a prior reading than it is currently, I know how many bytes have been added to the file and from there, I know the number of bytes I need to read from the end of the file and return to the console. All I need is a looping routine to constantly check the file for changes. In my code sample below, I emulate "tail -f" to the console by first reading the last 10 lines of a hypothetical BlackBerry Enterprise Server Management Agent log file (on an active server this file grows constantly) on a remote server as Unix "tail -f" would then start to monitor the log file for changes by comparing the file size waiting 100 milliseconds between each comparison. This process will continue until you send a break or the file is deleted.

With this bit of starter code, you should be able to implement some unique tools to monitor and react to data in log files in real-time.

UPDATE: I have made some improvements to this code here.

Function Read-EndOfFileByByteChunk($fileName,$totalNumberOfLines,$byteChunk) {
 if($totalNumberOfLines -lt 1) { $totalNumberOfLines = 1 }
 if($byteChunk -le 0) { $byteChunk = 10240 }
 $linesOfText = New-Object System.Collections.ArrayList
 if([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $asciiEncoding = New-Object System.Text.ASCIIEncoding
  $fileSize = $fileStream.Length
  $byteOffset = $byteChunk
  [byte[]] $bytesRead = New-Object byte[] $byteChunk
  $totalBytesProcessed = 0
  $lastReadAttempt = $false
  do {
   if($byteOffset -ge $fileSize) {
    $byteChunk = $fileSize - $totalBytesProcessed
    [byte[]] $bytesRead = New-Object byte[] $byteChunk
    $byteOffset = $fileSize
    $lastReadAttempt = $true
   }
   $fileStream.Seek((-$byteOffset), [System.IO.SeekOrigin]::End) | Out-Null
   $fileStream.Read($bytesRead, 0, $byteChunk) | Out-Null
   $chunkOfText = New-Object System.Collections.ArrayList
   $chunkOfText.AddRange(([System.Text.RegularExpressions.Regex]::Split($asciiEncoding.GetString($bytesRead),"\r\n")))
   $firstLineLength = ($chunkOfText[0].Length)
   $byteOffset = ($byteOffset + $byteChunk) - ($firstLineLength)
   if($lastReadAttempt -eq $false -and $chunkOfText.count -lt $totalNumberOfLines) {
    $chunkOfText.RemoveAt(0)
   }
   $totalBytesProcessed += ($byteChunk - $firstLineLength)
   $linesOfText.InsertRange(0, $chunkOfText)
  } while($totalNumberOfLines -ge $linesOfText.count -and $lastReadAttempt -eq $false -and $totalBytesProcessed -lt $fileSize)
  $fileStream.Close()
  if($linesOfText.count -gt 1) {
   $linesOfText.RemoveAt($linesOfText.count-1)
  }
  $deltaLines = ($linesOfText.count - $totalNumberOfLines)
  if($deltaLines -gt 0) {
   $linesOfText.RemoveRange(0, $deltaLines)
  }
 } else {
  $linesOfText.Add("[ERROR] $fileName not found") | Out-Null
 }
 return $linesOfText
}
#--------------------------------------------------------------------------------------------------#
Function Read-FileUpdates($fileName,$startSize) {
 $asciiEncoding = New-Object System.Text.ASCIIEncoding
 while([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $currentFileSize = $fileStream.Length
  if($currentFileSize -gt $startSize) {
   $byteChunk = $currentFileSize - $startSize
   [byte[]] $bytesRead = New-Object byte[] $byteChunk
   $fileStream.Seek((-$byteChunk), [System.IO.SeekOrigin]::End) | Out-Null
   $fileStream.Read($bytesRead, 0, $byteChunk) | Out-Null
   Write-Host ($asciiEncoding.GetString($bytesRead)) -noNewLine
   $startSize = $currentFileSize
  }
  $fileStream.Close()
  Start-Sleep -milliseconds 100
 }
}
#--------------------------------------------------------------------------------------------------#
Set-Variable -name inputFile -option Constant -value "\\japan-bes.ad.mydomain.local\E$\Program Files\Research In Motion\BlackBerry Enterprise Server\Logs\20110418\JAPAN-BES_MAGT_02_20110418_0001.txt"
#--------------------------------------------------------------------------------------------------#
if([System.IO.File]::Exists($inputFile)) {
 Write-Host (@(Read-EndOfFileByByteChunk $inputFile 10 1280) | Out-String) -noNewLine
 $fileStream = New-Object System.IO.FileStream($inputFile,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
 $fileSize = $fileStream.Length
 $fileStream.Close()
 Read-FileUpdates $inputFile $fileSize
} else {
 Write-Host "Could not find $inputFile..." -foregroundColor Red
}

MD5 Hash For PowerShell

For a project, I had a requirement to move selected files between two servers and provide proof that the copied matched exactly the original file. A common method to provide this proof is through a MD5 hash for each of the files to create a fingerprint. A MD5 hash provides the ability to compare two files without having to go to the trouble and slowness of comparing the files byte by byte using the algorithm created by Ron Rivest in 1991 to create a 128-bit hash. You can save the hash and use it for future reference to see if the file has changed.

In the code sample below, I use the System.Security.Cryptography.MD5CryptoServiceProvider class to create the hash in bytes then convert it to a string using the System.Bitconverter class and removing all dashes to create a 32 character representation of that hash. If you require even more bits to be present in your hash, the System.Security.Cryptography namespace provides up to 512 bits. To achieve this level of security, use System.Security.Cryptography.SHA512Managed class instead of System.Security.Cryptography.MD5CryptoServiceProvider in the function.

To test out this code, change out the file names below. Copy a file to a new location, run the function and see the results. Then open the copied file, make a minor edit and test again. It's interesting to see the changes in the returned hash.

Function Get-MD5Hash($fileName) {
 if([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $MD5Hash = New-Object System.Security.Cryptography.MD5CryptoServiceProvider
  [byte[]]$fileByteChecksum = $MD5Hash.ComputeHash($fileStream)
  $fileChecksum = ([System.Bitconverter]::ToString($fileByteChecksum)).Replace("-","")
  $fileStream.Close()
 } else {
  $fileChecksum = "ERROR: $fileName Not Found"
 }
 return $fileChecksum
}

$fileName = "\\east-coast-fs.ad.mycompany.local\documents\Important Excel Spreadsheet.xlsx"
$fileChecksumOne = Get-MD5Hash $fileName
if($fileChecksumOne -match "^ERROR:") {
 Write-Host $fileChecksumOne -foregroundColor Red
 exit
}

$fileName = "\\west-coast-fs.ad.mycompany.local\documents\Important Excel Spreadsheet.xlsx"
$fileChecksumTwo = Get-MD5Hash $fileName
if($fileChecksumTwo -match "^ERROR:") {
 Write-Host $fileChecksumTwo -foregroundColor Red
 exit
}

if($fileChecksumOne -eq $fileChecksumTwo) {
 Write-Host "Files match!"
 Write-Host $fileChecksumOne -foregroundColor Green
 Write-Host $fileChecksumTwo -foregroundColor Green
} else {
 Write-Host "Files do not match!"
 Write-Host $fileChecksumOne -foregroundColor Red
 Write-Host $fileChecksumTwo -foregroundColor Green
}

Unix Tail-like Functionality in PowerShell Revisited

My first attempt to replicate tail for PowerShell, which I wrote about in "Unix Tail-like Functionality in PowerShell", was horribly inefficient once you got past a couple dozen lines. This makes since given the method I was using -- a byte by byte reverse read of a text file, converting each byte at a time to ASCII. I knew the solution was to "wolf down" large byte chunks and process them as a whole. Using System.Text.ASCIIEncoding.GetString, I am doing that just after reading into memory multiple bytes using System.IO.FileStream.Read. With this change in methodology, I am getting to within 3% of the speed of tail in UNIX in my tests. The largest test I've performed was returning 1,000,000 lines from a 850MB log file. A Mac OS X 10.6.6 workstation performed the task in 16 seconds using tail and a Windows Server 2003 server returned in 17 seconds using this method. Good enough for me. Most of my needs are in the thousands of lines which I am able to return in hundreds of milliseconds which is perfect my monitoring scripts in Nagios. Compared to my previous attempt, this is a Lockheed SR-71 vs. a Wright Brothers Flyer. A small 5,000 tail using the old code took 5 1/2 minutes to return while this code returned the same lines in 200 milliseconds. Huge difference!

In the code sample below, I am using 10 kilobytes for that chunking. I found that number suited most of my needs. However, you can greatly increase that number for large number of lines to be returned (I used 4MB for my million line test). You can also do a little automatic tuning by altering the number of bytes using the number of lines you are seeking. One thing to be aware when passing files to this code, if you pass a file to System.IO.File/FileStream without a full path, it will not assume the file is located in the path of the executed script so Test-Path is not a valid test. Using System.IO.Directory.GetCurrentDirectory, you can find this by running the following in PowerShell:

[System.IO.Directory]::GetCurrentDirectory()

More than likely, it will point to the home directory of the profile the shell is executed under.

Also be aware that this tail-like function does not handle unicode log files. The method I am using to decode the bytes is ASCII dependent. I am not using System.Text.UnicodeEncoding yet in the code. Currently ASCII meets all my needs for reading log files but I am still interested in adding compatibility to this function. I am also assuming that all log files denote the end of a line using carriage return & line feed (CHR 13 + CHR 10) which is how majority of text files are written in Windows. UNIX & old style Macintosh text files will not work properly with this code. You will need to modify line 23 to change the delimiter for the split for those text file formats.

UPDATE: I have now finished an update that provides the "tail -f" functionality for continuously reading the updates to a text file. Read about it in my blog post, Replicating UNIX "tail -f" in PowerShell.

UPDATE: I have updated the code to handle unicode text files and non-Windows new lines. You can review the code here.

Function Read-EndOfFileByByteChunk($fileName,$totalNumberOfLines,$byteChunk) {
 if($totalNumberOfLines -lt 1) { $totalNumberOfLines = 1 }
 if($byteChunk -le 0) { $byteChunk = 10240 }
 $linesOfText = New-Object System.Collections.ArrayList
 if([System.IO.File]::Exists($fileName)) {
  $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
  $asciiEncoding = New-Object System.Text.ASCIIEncoding
  $fileSize = $fileStream.Length
  $byteOffset = $byteChunk
  [byte[]] $bytesRead = New-Object byte[] $byteChunk
  $totalBytesProcessed = 0
  $lastReadAttempt = $false
  do {
   if($byteOffset -ge $fileSize) {
    $byteChunk = $fileSize - $totalBytesProcessed
    [byte[]] $bytesRead = New-Object byte[] $byteChunk
    $byteOffset = $fileSize
    $lastReadAttempt = $true
   }
   $fileStream.Seek((-$byteOffset), [System.IO.SeekOrigin]::End) | Out-Null
   $fileStream.Read($bytesRead, 0, $byteChunk) | Out-Null
   $chunkOfText = New-Object System.Collections.ArrayList
   $chunkOfText.AddRange(([System.Text.RegularExpressions.Regex]::Split($asciiEncoding.GetString($bytesRead),"\r\n")))
   $firstLineLength = ($chunkOfText[0].Length)
   $byteOffset = ($byteOffset + $byteChunk) - ($firstLineLength)
   if($lastReadAttempt -eq $false -and $chunkOfText.count -lt $totalNumberOfLines) {
    $chunkOfText.RemoveAt(0)
   }
   $totalBytesProcessed += ($byteChunk - $firstLineLength)
   $linesOfText.InsertRange(0, $chunkOfText)
  } while($totalNumberOfLines -ge $linesOfText.count -and $lastReadAttempt -eq $false -and $totalBytesProcessed -lt $fileSize)
  $fileStream.Close()
  if($linesOfText.count -gt 1) {
   $linesOfText.RemoveAt($linesOfText.count-1)
  }
  $deltaLines = ($linesOfText.count - $totalNumberOfLines)
  if($deltaLines -gt 0) {
   $linesOfText.RemoveRange(0, $deltaLines)
  }
 } else {
  $linesOfText.Add("[ERROR] $fileName not found") | Out-Null
 }
 return $linesOfText
}
#--------------------------------------------------------------------------------------------------#
$fileName = "C:\Logs\really-huge.log" # Your really big log file
$numberOfLines = 100 # Number of lines from the end of the really big log file to return
$byteChunk = 10240 # Size of bytes read per seek during the search for lines to return
####################################################################################################
## This is a possible self-tuning method you can use but will blow up memory on an enormous 
## number of lines to return
## $byteChunk = $numberOfLines * 256 
####################################################################################################
$lastLines = @()

$lastLines = Read-EndOfFileByByteChunk $fileName $numberOfLines $byteChunk
foreach($lineOfText in $lastLines) {
 Write-Output $lineOfText
}

Unix Tail-like Functionality in PowerShell

A common tool I use in shell scripts on Unix/Linux/Mac OS X servers is tail. While there are command-line tail conversions for Windows, I need something I can integrate into a script for reading the end of large log files, search for information and act on that result. I don't want to distribute third party software along with the script to accomplish the task. Get-Content and Select-Object are not suitable for large files.

After researching the capabilities of File IO in .Net, I found that System.IO.FileStream class had just what I needed. Using this class, I read the target text file byte by byte from the end of the file until I reach a selected number of lines of text delimited by a carriage return. The amount of time it takes to obtain the data is related to the number of characters per line. It works very well in 500 lines or less in my typical log files (I tested up to 1 gigabyte) and much faster than using:

Get-Content "C:\Logs\really-huge.log" | Select-Object -last 100

The code meets 95% of my needs but I am sure I can optimize it so it comes close to matching the speed of tail from the Unix distributions I commonly use. It's my first stab at tackling the problem. One interesting part of the code is that I use System.Collections.ArrayList instead of a standard PowerShell array. The reason is since I am reading the file in reverse, I need to return the data back in the proper order. The ArrayList object allows me to insert into the first element so I don't have to re-write the array in the right order after collecting the data. Also I noticed that using System.Convert to covert the bytes to a character instead of using PowerShell's native [char] was faster. Returning large number of lines, it was significant -- about .5 seconds per 100 lines.

I will keep working on this to close that 5% and update this post with a link to an updated blog post in the future with the improvements.

UPDATE: I have rewritten this function in a new blog post and it is lightning fast. This code is deprecated and should only be used for amusement purposes.

Function Read-EndOfFile($fileName,$totalNumberOfLines) {
 $fileStream = New-Object System.IO.FileStream($fileName,[System.IO.FileMode]::Open,[System.IO.FileAccess]::Read,[System.IO.FileShare]::ReadWrite)
 $linesOfText = New-Object System.Collections.ArrayList
 $byteOffset = 1
 $lineOfText = ""
 do {
   $fileStream.Seek((-$byteOffset), [System.IO.SeekOrigin]::End) | Out-Null
  $byte = $fileStream.ReadByte()
  if($byte -eq 13) {
  } elseif($byte -eq 10) {
   $linesOfText.Insert(0, $lineOfText)
   $lineOfText = ""
  } else {
   $lineOfText = [System.Convert]::ToChar($byte) + $lineOfText
  }
  $byteOffset++
 } while ($linesOfText.count -le $totalNumberOfLines)
 $fileStream.Close()
 return $linesOfText
}
#--------------------------------------------------------------------------------------------------#
$fileName = "C:\Logs\really-huge.log" # Your really big log file
$numberOfLines = 100 # Number of lines from the end of the really big log file to return

if([System.IO.File]::Exists($fileName) -and $numberOfLines -gt 0) {
 $lastLines = Read-EndOfFile $fileName $numberOfLines

 foreach($lineOfText in $lastLines) {
  Write-Output $lineOfText
 }
}

File System Rights -- Part 2

In this code example, I expand upon the NTFS rights exercise in the my blog post, "File System Rights -- Part 1". We use the same bitwise and operation to determine rights. This version, however, provides the same information found in the advance tab of Security for the "allow". I will demonstrate in later posts how to determine if a "deny" is selected. This script returns a much better report to provide to the non-technical user than the information recorded in Part 1.

Now that you have this information regarding NTFS permissions for a file system object, you can do some interesting things. You can monitor for changes. Alter permissions based on logic derived from the results returned. Clone permissions between objects. Consolidate permissions in defined access groups to reduce access bloat and speed up access -- I find this on non-Windows based NAS devices all the time. All of these topics I will provide examples in future blog posts.

Function Get-AdvancedPermissions($fileSystemRights) {
 $permissions = @()
 
 if ($fileSystemRights -band 0x1 -and $fileSystemRights -band 0x1 -and $fileSystemRights -band 0x2 -and $fileSystemRights -band 0x2 -and $fileSystemRights -band 0x4 -and $fileSystemRights -band 0x4 -and $fileSystemRights -band 0x8 -and $fileSystemRights -band 0x10 -and $fileSystemRights -band 0x20 -and $fileSystemRights -band 0x20 -and $fileSystemRights -band 0x40 -and $fileSystemRights -band 0x80 -and $fileSystemRights -band 0x100 -and $fileSystemRights -band 0x116 -and $fileSystemRights -band 0x10000 -and $fileSystemRights -band 0x20000 -and $fileSystemRights -band 0x20089 -and $fileSystemRights -band 0x200a9 -and $fileSystemRights -band 0x301bf -and $fileSystemRights -band 0x40000 -and $fileSystemRights -band 0x80000 -and $fileSystemRights -band 0x100000 -and $fileSystemRights -band 0x1f01ff) {
  $permissions += "Full Control"
 } 
 
 if ($fileSystemRights -band 0x1 -and  $fileSystemRights -band 0x10 -and  $fileSystemRights -band 0x100 -and  $fileSystemRights -band 0x10000 -and  $fileSystemRights -band 0x100000 -and  $fileSystemRights -band 0x116 -and  $fileSystemRights -band 0x2 -and  $fileSystemRights -band 0x20 -and  $fileSystemRights -band 0x20000 -and  $fileSystemRights -band 0x20089 -and  $fileSystemRights -band 0x200a9 -and  $fileSystemRights -band 0x301bf -and  $fileSystemRights -band 0x4 -and  $fileSystemRights -band 0x8 -and  $fileSystemRights -band 0x80) {
  $permissions += "Modify"
 }

 if ($fileSystemRights -band 0x200a9 -and $fileSystemRights -band 0x20 -and $fileSystemRights -band 0x1 -and $fileSystemRights -band 0x80 -and $fileSystemRights -band 0x8 -and $fileSystemRights -band 0x20000) { # -and $fileSystemRights -band 0x1 -and $fileSystemRights -band 0x80 -and $fileSystemRights -band 0x80x8 -and $fileSystemRights -band 0x20000) {
  $permissions += "Read & Execute"
 }

 if ($fileSystemRights -band 0x1 -and $fileSystemRights -band 0x1 -and $fileSystemRights -band 0x8 -and $fileSystemRights -band 0x20 -and $fileSystemRights -band 0x20000) {
  $permissions += "List Folder Contents"
 }

 if ($fileSystemRights -band 0x20089) { $permissions += "Read" }
 if ($fileSystemRights -band 0x116) { $permissions += "Write" }
 if ($fileSystemRights -band 0x20) { $permissions += "Traverse Folder / Execute File" }
 if ($fileSystemRights -band 0x1) { $permissions += "List Folder / Read Data" }
 if ($fileSystemRights -band 0x80) { $permissions += "Read Attributes" }
 if ($fileSystemRights -band 0x8) { $permissions += "Read Extended Attriibutes" }
 if ($fileSystemRights -band 0x2) { $permissions += "Create Files / Write Data" }
 if ($fileSystemRights -band 0x4) { $permissions += "Create Folders / Append Data" }
 if ($fileSystemRights -band 0x100) { $permissions += "Write Attributes" }
 if ($fileSystemRights -band 0x10) { $permissions += "Write Extended Attributes" }
 if ($fileSystemRights -band 0x40) { $permissions += "Delete Subfolders and Files" }
 if ($fileSystemRights -band 0x10000) { $permissions += "Delete" }  
 if ($fileSystemRights -band 0x20000) { $permissions += "Read Permissions" }
 if ($fileSystemRights -band 0x40000) { $permissions += "Change Permissions" }
 if ($fileSystemRights -band 0x80000) { $permissions += "Take Ownership" }

 if($fileSystemRights -band 0x10000000) {
  $permissions += "Full Control"
  $permissions += "Modify"
  $permissions += "Read & Execute"
  $permissions += "List Folder Contents"
  $permissions += "Write"
  $permissions += "Traverse Folder / Execute File"
  $permissions += "List Folder / Read Data"
  $permissions += "Read Attributes"
  $permissions += "Read Extended Attriibutes" 
  $permissions += "Create Files / Write Data"
  $permissions += "Create Folders / Append Data"
  $permissions += "Write Attributes"
  $permissions += "Write Extended Attributes"
  $permissions += "Delete Subfolders and Files"
  $permissions += "Delete"
  $permissions += "Read Permissions"
  $permissions += "Change Permissions"
  $permissions += "Take Ownership"
 }

 return $permissions
}

$uncPath = "\\server.ad.mycompany.local\share\directory"
#$uncPath = "\\server.ad.mycompany.local\share\directory\file.txt"
if(Test-Path -path $uncPath) {
 Write-Host $uncPath -foregroundColor Yellow
 $acl = Get-Acl -path $uncPath
 $aces = $acl.GetAccessRules($true, $true, [System.Security.Principal.NTAccount])
 foreach($ace in $aces) {
  $identityReference = $ace.IdentityReference.Value
  Write-Host $identityReference -foregroundColor Green
  $permissions = Get-AdvancedPermissions $ace.FileSystemRights
  foreach($permission in $permissions) {
   Write-Host "`t$permission"
  }
  Write-Host ("-" * 70)
 }
}

Validating Command Line Parameters

A common reason I write scripts is to provide other members of my IT community the ability to perform functions that standard tools cannot satisfy. Many of these scripts require multiple parameters in order to return the desired result. The last thing I want to do is take to time to explain why something I wrote failed to provide the desired result because of an input error. Human error when someone else is using your work should be forefront in your mind. Evaluating the parameters supplied to a script and making sure that they meet a certain level of sanity reduces those chances.

You can do basic checking directly in the param using Read-Host but that just insures a value is provided.

param([string]$path = $(Read-Host -prompt "`tPath"))

This will not guarantee that the data the script is expecting will be provided by the user nor can it provide confirmation that it is what the user meant. This can be very crucial when dealing with scripts that may delete or modify objects. Sometimes, you may even want to go the extra mile and ask, "Are you really sure?". The following code is a typical interrogation I would do when asking for a path to be supplied.

param([string]$path)

if(![bool]$path) {
 Write-Host "You are missing the `"-path`" command line argument" -foregroundColor Red
 Write-Host ""
 Write-Host "This is the path that the script requires to perform its function."
 Write-Host ""
 while((![bool]$path)) {
  $path = Read-Host -prompt "`tPath"
 }
 Write-Host ""
}
if(!(Test-Path -path $path)) {
 Write-Host "Unable to locate: $path" -foregroundColor Red
 Write-Host ""
 Write-Host "Please review the correct path"
 Write-Host ""
 while(!(Test-Path -path $path)) {
  $path = Read-Host -prompt "`tPath"
 }
 Write-Host ""
}

$answer = $null
while($answer -ne "Y" -and $answer -ne "N") {
 $answer = Read-Host -prompt "Are you sure you want to use $path (Y/N)?"
}

if($answer -eq "N") {
 Write-Host ""
 Write-Host "Quiting..." -foregroundColor Yellow
 exit
}

Write-Host ""
Write-Host "`$path = $path"

File System Rights -- Part 1

A common request I receive is "Who has rights to this?". To know this, we need to know the access rights and access masks for the securable object. The access mask is a 32 bit value that we can determine the access rights. To obtain this information in PowerShell, we use the Get-Acl cmdlet to obtain the security of the file system object. In the code below, we use Get-Acl and query the access rules and perform bitwise and operations to obtain the masks for each identity. This is a good first start in understanding how to return the information for "Who has rights to this?" but that return might not be easily digestible to the typical user. In a future, I will supply a function that will return exactly what appears in the advanced permissions tab on a file or folder -- a little more english than techish.

Function Get-Permissions($fileSystemRights) {

 $permissions = @()

# Read access
 if($fileSystemRights -band 0x80000000) { $permissions += "GENERIC_READ" }
 
# Write access
 if($fileSystemRights -band 0x40000000) { $permissions += "GENERIC_WRITE" }
 
# Execute access
 if($fileSystemRights -band 0x20000000) { $permissions += "GENERIC_EXECUTE" }
 
# Read, write, and execute access
 if($fileSystemRights -band 0x10000000) { $permissions += "GENERIC_ALL" }

 if($fileSystemRights -band 0x8000000) { $permissions += "RESERVED_1" }
 if($fileSystemRights -band 0x4000000) { $permissions += "RESERVED_2" }
 if($fileSystemRights -band 0x2000000) { $permissions += "MAXIMUM_ALLOWED" }
 if($fileSystemRights -band 0x1000000) { $permissions += "ACCESS_SYSTEM_SECURITY" }
 if($fileSystemRights -band 0x800000) { $permissions += "UNKNOWN" }
 if($fileSystemRights -band 0x400000) { $permissions += "UNKNOWN" }
 if($fileSystemRights -band 0x200000) { $permissions += "UNKNOWN" }
 
# Essentially Full Control
 if($fileSystemRights -band 0x1F01FF) { $permissions += "FILE_ALL_ACCESS" }
 
# Combines DELETE, READ_CONTROL, WRITE_DAC, WRITE_OWNER, and SYNCHRONIZE access.
 if($fileSystemRights -band 0x1F0000) { $permissions += "STANDARD_RIGHTS_ALL" }

# Combines DELETE, READ_CONTROL, WRITE_DAC, and WRITE_OWNER access. 
 if($fileSystemRights -band 0xF0000) { $permissions += "STANDARD_RIGHTS_REQUIRED" }

# Read
 if($fileSystemRights -band 0x120089) { $permissions += "FILE_GENERIC_READ" }
 if($fileSystemRights -band 0x12010E) { $permissions += "FILE_GENERIC_WRITE" }
 if($fileSystemRights -band 0x1200A0) { $permissions += "FILE_GENERIC_EXECUTE" }
 
# Currently defined to equal READ_CONTROL.
 if($fileSystemRights -band 0x20000) { $permissions += "STANDARD_RIGHTS_READ" }
 
# Currently defined to equal READ_CONTROL.
 if($fileSystemRights -band 0x20000) { $permissions += "STANDARD_RIGHTS_WRITE" }
 
# Currently defined to equal READ_CONTROL.
 if($fileSystemRights -band 0x20000) { $permissions += "STANDARD_RIGHTS_EXECUTE" }
 if($fileSystemRights -band 0xFFFF) { $permissions += "SPECIFIC_RIGHTS_ALL" }

#  permission to synchronize
#  bit indicating permission to perform synchronize operation, used sometimes during file access
#  this permission is automaticaly granted with read and write access and revoked when read or write access is denied. 
#  it is not displayed in the list of permissions in Windows UI
 if($fileSystemRights -band 0x100000) { $permissions += "SYNCHRONIZE" }
 
#  permission to assign owner
#  corresponds to Take Ownership permissions in Windows UI
 if($fileSystemRights -band 0x80000) { $permissions += "WRITE_OWNER" }
 
#  permission to change discretionary ACL
#  corresponds to Change Permissions permissions in Windows UI
 if($fileSystemRights -band 0x40000) { $permissions += "WRITE_DAC" }
 
#  permission to read security descriptor
#  corresponds to Read Permissions permissions in Windows UI
 if($fileSystemRights -band 0x20000) { $permissions += "READ_CONTROL" }
 
#  permission to delete file or folder
#  corresponds to Delete permissions in Windows UI
 if($fileSystemRights -band 0x10000) { $permissions += "DELETE" }
 if($fileSystemRights -band 0x10000) { $permissions += "FILE_DELETE" }
 
#  permission to change file or folder attributes
#  corresponds to Write Attributes permissions in Windows UI
 if($fileSystemRights -band 0x100) { $permissions += "FILE_WRITE_ATTRIBUTES" }
 
#  permission to read file or folder attributes
#  corresponds to Read Attributes permissions in Windows UI
 if($fileSystemRights -band 0x80) { $permissions += "FILE_READ_ATTRIBUTES" }
 
#  permission to delete directory and all files it contains
#  corresponds to Delete Subfolders and Files permissions in Windows UI
 if($fileSystemRights -band 0x40) { $permissions += "FILE_DELETE_CHILD" }

#  permission to execute file or traverse directory
#  corresponds to Traverse Folder / Execute File permissions in Windows UI
 if($fileSystemRights -band 0x20) { $permissions += "FILE_EXECUTE" }
 if($fileSystemRights -band 0x20) { $permissions += "FILE_TRAVERSE" }
 
#  permission to write extended attributes
#  corresponds to Write Extended Attributes permissions in Windows UI
 if($fileSystemRights -band 0x10) { $permissions += "FILE_WRITE_EA" }
 
#  permission to read extended attributes
#  corresponds to Read Extended Attributes permissions in Windows UI
 if($fileSystemRights -band 0x8) { $permissions += "FILE_READ_EA" }

#  permission to append data to file or to create subdirectory
#  corresponds to Create Folders / Append Data permissions in Windows UI
 if($fileSystemRights -band 0x4) { $permissions += "FILE_APPEND_DATA" }
 if($fileSystemRights -band 0x4) { $permissions += "FILE_ADD_SUBDIRECTORY" }
 
#  permission to write data to file or create file in directory
#  corresponds to Create Files / Write Data permissions in Windows UI
 if($fileSystemRights -band 0x2) { $permissions += "FILE_WRITE_DATA" }
 if($fileSystemRights -band 0x2) { $permissions += "FILE_ADD_FILE" }
 
#  permission to read data from file or list contents of directory
#  corresponds to List Folder / Read Data permissions in Windows UI
 if($fileSystemRights -band 0x1) { $permissions += "FILE_READ_DATA" }
 if($fileSystemRights -band 0x1) { $permissions += "FILE_LIST_DIRECTORY" }

 return $permissions
}

$uncPath = "\\server.ad.mycompany.local\share\directory"
#$uncPath = "\\server.ad.mycompany.local\share\directory\file.txt"
if(Test-Path -path $uncPath) {
 Write-Host $uncPath -foregroundColor Yellow
 $acl = Get-Acl -path $uncPath 
 $aces = $acl.GetAccessRules($true, $true, [System.Security.Principal.NTAccount])
 foreach($ace in $aces) {
  $identityReference = $ace.IdentityReference.Value
  Write-Host $identityReference -foregroundColor Green
  $permissions = Get-Permissions $ace.FileSystemRights
  foreach($permission in $permissions) {
   Write-Host "`t$permission"
  }
  Write-Host ("-" * 70)
 }
}

Searching for Files in Nested Directories

Sometimes I need more fine grained control over searching for files in nested directories than the "-recurse" option for Get-ChildItem provides or I don't want to write some extremely long piped command to achieve the desired result. Below is a code snippet that is a good jumping off point to achieving that goal. This snippet will search the C: drive for Office documents and text files and output them to a comma separated value file. You could swap the local path to a UNC path and search a network share.

Function Search-NestedDirectory($path,$files) {
 foreach($childItem in (Get-ChildItem -literalPath $path -force | Sort-Object -property FullName)) {
  if($childItem.PSIsContainer -eq "True") {
   Write-Host ("Scanning: " + $childItem.FullName) -foregroundColor Green
   if($files.count -ge 1) {
    Write-Host ("Files Found: " + $files.count)
   }
   $files = Search-NestedDirectory $childItem.FullName @($files)
  } elseif($childItem.Extension -eq ".xls" -or $childItem.Extension -eq ".doc" -or $childItem.Extension -eq ".ppt" -or $childItem.Extension -eq ".txt") {
   $file = New-Object -typeName PSObject
   Add-Member -inputObject $file -type NoteProperty -name "path" -value $childItem.FullName
   Add-Member -inputObject $file -type NoteProperty -name "name" -value $childItem.Name
   Add-Member -inputObject $file -type NoteProperty -name "type" -value $childItem.Extension
   Add-Member -inputObject $file -type NoteProperty -name "bytes" -value $childItem.Length
   Add-Member -inputObject $file -type NoteProperty -name "creationTime" -value ($childItem.CreationTime).ToString()
   $files += $file
  }
 }
 return $files
}

$files = Search-NestedDirectory "C:\" @()

$files | Export-Csv -path "files.csv" -noTypeInformation