23 July 2015

Extract links froma web site and download multiple files with Poweshell Script

Few weeks ago, I've posted about how to download multiple files from web using powershell. In this post I'm going to show you a script that retrieves the links from a given web site and downloads the files to a given directory on your local machine.



#place to store downloaded files
$filepath = Read-Host "Please enter the location to store the files.(eg. C:\Downloads\)"
$webSite = Read-Host "Please enter the site url that contains the links: (eg. https://www.example.com/photos)"
#gathering link details from source web
$links = ((Invoke-WebRequest Uri $webSite).Links)
#downloading...
foreach($link in $links) {
#link to files
$item = $webSite + $link.href
Write-Host Url: $item
# innertext contains the file name
Write-Host Title:  $link.innerText
#download path on local machine
$path = $filepath + $link.innerText
Write-Host $path
#downloading files
Start-BitsTransfer $item  $path

}

08 July 2015

IIS error when installing SharePoint 2013 on Windows Server 2012 R2 : The tool was unable to install Application Server Role, Web Server (IIS) Role


This error has cost me five or six hours. Then I tried everything from applying patches, trying to install IIS manually, etc. Once I figured it out, I installed SharePoint Server and went with my life happily.
It happened to one of my colleague and I couldn't remember what fix I did a month ago. Later, I figured it out and solved. I'm documenting it here so that if anyone come across the same issue, they can solve it.

This error occurred because the pre-request installer is trying to execute ServerManagerCmd.exe from system 32 folder and failing. (I found out this from the pre-request installer logs). So I get in to the system32 folder to see what this fuss about. Voila! There was no ServerManagerCmd.exe in the folder, instead there was a ServerManager.exe. Looks like the executable file was renamed in WS 2012 R2.

So, I made duplicate copy of ServerManager.exe by copy and pasting in the same folder(system32), then renamed the copy as  ServerManagerCmd.exe. Ran the pre-request installer again and installed successfully.

Summary

Issue: The tool was unable to install Application Server Role, Web Server (IIS)

Solution:
                Step 1: Navigate to system32 folder (C:\Windows\system32)
                Step 2: Copy ServerManager.exe and paste in the same location. (system32 folder)
                Step 3: Rename ServerManager-Copy.exe to ServerManagerCmd.exe
                Step 4: Run the pre-request installer

By the way, SharePoint 2013 with SP1 works fine without any workarounds. So, If you have Windows Server 2012 R2, it is recommended to install SharePoint 2013 with SP1.

24 April 2015

Download Multiple files from web using powershell

Today I've come across a task that requires 1000 of files to be downloaded from a web directory. Now I have to click the link, then when save as option shows up click again to save. That would require 2000+ mouse clicks. No way. So I took powershell, and made a script to do it for me.

First I saved the Url and file name is a csv file. Csv file looks like
Url, Title
http://example.com/example/SomeFile.jpg, SomeFile.jpg
http://example.com/example/AnotherFile.jpg, AnotherFile.jpg

Now how I get this? I just right click the page, go to source then copied all to my favorite text editor Notepad++. After that did some Find & Replace to remove the html elements. (replace


  • with "" ).


  • Anyway, here is the script

     
    #csv file to compare

    $csv = IMPORT-CSV C:\R\picts.csv
    $filepath = "C:\Users\janbalagan\Downloads\Picts\Collection\"
    foreach($item in $csv)
    {
    $path = $filepath + $item.Title
    Write-Host "Path: " $path
     Write-Host $item.Title " being downloaded: "
    Invoke-WebRequest $item.Url -OutFile $path
    }
    Write-Host "Download Completed" -BackgroundColor DarkRed


    In case if you are working with powershell 2.0 which comes with windows 7 and previous versions, you must use 
    (New-Object Net.WebClient).DownloadFile($item.Url, $path) instead of
    Invoke-WebRequest $item.Url -OutFile $path

    19 April 2015

    Compare and Update Excel spread sheet using powershell

    Windows Power-shell is the most useful tool to automate lot of our work. It comes installed with Windows OS. Last week I've got some task where I have to compare some data from a csv file to an excel spread sheet and update the relevant data. Excel sheet contained thousands of records which has to be compared with a csv file contained hundreds of records. I have got more than 20 sets of files to compare and update.
    As a programmer, I hate this kind of mundane tasks. Those are ultimately boring. So, I came up with a power-shell script to do the job for me. This script can be modified to compare and update between two or more excel spread sheets as well. Feel free to modify and use it.

    # excel file path
    $excel_file_path = "C:\R\All Sites for migration 4-16.xlsx";
    #csv file to compare
    $csv = IMPORT-CSV C:\R\16_4_2015_B.csv
    
    # date completed to update excel column
    $dateNow = "17/4"
    # migration status to update excel column
    $status = "TEST STATUS"
    
    # Instantiate the COM object
    $Excel = New-Object -ComObject Excel.Application
    # Opens the excel workbook
    $ExcelWorkBook = $Excel.Workbooks.Open($excel_file_path);
    # prepare the work sheet. Here 1 means the first worksheet
    # you can use the worksheet name as well. eg: Sheet 1
    $ExcelWorkSheet = $Excel.WorkSheets.item(1)
    
    
    Write-host "Reading Excel File" -NoNewline
    # row number. I set this as 2 because I'm goint to read from second row
    # as first row contains the heading
    $intRow = 2;
    # starting the loop
    Do
    {
    #reading from cell. First parametrer $intRow is the row being read
    # second parameter 3 gets the third column from the spread sheet
    $readValue = $ExcelWorkSheet.Cells.Item($intRow,3).value2
    # traversing through csv file contents to compare with the selected row
    foreach($item in $csv)
    {
     # if third column value matches with csv file Url
     if($item.Url -eq $readValue)
     {
      # writing the match on screen
      Write-Host "Match:" $readValue "DB:" $item.DBSize -foregroundcolor "Magenta"
         # updating data to excel file columns 5,6 and 9
        $ExcelWorkSheet.Cells.Item($intRow,5) = $dateNow
        $ExcelWorkSheet.Cells.Item($intRow,6) = $status
        $ExcelWorkSheet.Cells.Item($intRow,9) = $item.DBSize
     }
    }
    # increment row to the next row
    $intRow ++
    }While ($ExcelWorkSheet.Cells.Item($intRow,1).value2 -ne $null) # loop untill there is no value in the first column
    # Saving the file
    $ExcelWorkBook.Save();
    # Closing Work book
    $ExcelWorkBook.Close();
    # Closing Excel
    $Excel.Quit();
    # Releasing the com object
    [System.Runtime.Interopservices.Marshal]::ReleaseComObject($Excel)
    Stop-Process -Name EXCEL -Force

    13 January 2012

    Surf Internet Anonymously and break the Country Barriers for Contents

    Guys,
    Today I would like to give you an introduction about TOR network. You may think what make me to write this post today. Wel, Indian authorities warned facebook and google that they gonna block and shutdown their sites as China did, if they allow the content to users which government doesn't want their citizens know.. (But they are naming those as abusive contents.) But, my personal opinion is, every person has the right to know the information. And everyone has the sense to decide which are true and which are not.

    On other hand, there are already barriers on country based, community based, etc. But we, as computer geeks, need not to worry about those things. We always have an alternative way to get our things done.

    TOR network helps users to surf internet anonymously and encrypt all the data flow through the network except the last exit point. Which means no one can make out what page you gonna browse or what data you are sending through, not even your Internet Service Provider.

    Before I give you a link to download, I would like to explain how TOR works.



    FROM WIKI:-


    Tor (short for The onion router) is a system intended to enable online anonymity. Tor client software routes Internet traffic through a worldwide volunteer network of servers in order to conceal a user's location or usage from anyone conducting network surveillance or traffic analysis. Using Tor makes it more difficult to trace Internet activity, including "visits to Web sites, online posts, instant messages and other communication forms", back to the user and is intended to protect users' personal freedom, privacy, and ability to conduct confidential business by keeping their internet activities from being monitored.


    "Onion routing" refers to the layered nature of the encryption service: The original data are encrypted and re-encrypted multiple times, then sent through successive Tor relays, each one of which decrypts a "layer" of encryption before passing the data on to the next relay and, ultimately, its destination. This reduces the possibility of the original data being unscrambled or understood in transit.

    A routing onion (or just onion) is a data structure formed by 'wrapping' a plaintext message with successive layers of encryption, such that each layer can be 'unwrapped' (decrypted) like the layer of an onion by one intermediary in a succession of intermediaries, with the original plaintext message only being viewable by at most:
    1. the sender
    2. the last intermediary
    3. the recipient
    If there is end-to-end encryption between the sender and the recipient, then not even the last intermediary can view the original message; this is similar to a game of 'pass the parcel'.
    An intermediary is traditionally called a node or router.
    To create and transmit an onion, the following steps are taken:
    1. The sender picks nodes from a list provided by a special node called the directory node (traffic between the sender and the directory node may also be encrypted or otherwise anonymised or decentralised); the chosen nodes are ordered to provide a path through which the message may be transmitted; this ordering of the nodes is called a chain or a circuit.
    2. Using asymmetric key cryptography, the sender uses the public key of each chosen node to wrap the plaintext message in the necessary layers of encryption: The public keys are retrieved from an advertised list or by on-the-spot negotiation for temporary use, and the layers are applied in reverse order of the message's path from sender to receiver; with each layer, the client includes information for the corresponding node regarding the next node to which the onion should be transmitted.
    3. As the onion passes to each node in the chain, a layer of encryption is peeled away by the receiving node (using the private key that corresponds to the public key with which the layer was encrypted), and then the newly diminished onion is transmitted to the next node in the chain.
    4. The last node in the chain peels off the last layer and transmits the original message to the intended recipient.
    Using this approach means each node in the chain is ideally aware of only 2 other nodes:
    1. the preceding node from which the onion was transmitted.
    2. the proceeding node to which the onion should next be transmitted.
    The peeling away of each layer of the onion makes it difficult or impossible to track the onion without compromising a significant number of nodes.

    Ok. Enough of information. Now I'll give you the link to TOR client. Download it. Play with it.. Ask us if you have any doubts or questions. You can download it from the official TOR website
    https://www.torproject.org/download/download-easy.html.en