Saturday, July 28, 2018

Simple Web Client, the Sequel

In an earlier post, A Simple Web Client, I demonstrated a simple program for pulling data from a download website to an Oracle table using the htp package.  A few years later, I needed to do a similar data pull to a SQL Server table.  In this post, I'll demonstrate a simple way of pulling data from an URL into a SQL Server table.

In an ideal world, languages and packages would be standard enough that one could readily port a stored procedure from one vendor's database and reuse the code without too much trouble in another vendor's database.    While most SQL is pretty transportable, the stored procedure languages are highly proprietary and do not port at all from one vendor to another ( one exception: IBM's support of Oracle's PL/SQL language).

The  SQL Server solution takes a different approach from the  Oracle PL/SQL solution.   In the latter,  an Oracle PL/SQL procedure used built-in Oracle  packages to open and manage an HTTP connection, to read data from the HTTP socket, and to write the data to the console.  In practice, instead of writing the data to the console, we might have processed the data  directly or loaded the data to a staging table for later processing.   The SQL Server solution is different; rather than use T-SQL to manage the HTTP connection, this solution uses an external program to read the data from the web into a file on a Windows Server before bulk importing the data  into a SQL Server staging table.  After we load the data to a staging table, we can process the data as needed by our application.

Let's look at the SQL Server solution.  First, we'll use the data from the Federal Reserve from the previous post again.  Put the URL https://www.federalreserve.gov/datadownload/Output.aspx?rel=H10&series=122e3bcb627e8e53f1bf72a1a09cfb81&lastObs=10&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package into your browser.   When your browser opens the URL, you'll get a CSV file like the following:

Downloaded data opened in a spreadsheet

We have a four-column spreadsheet, with some meta-data in rows 1 through 5, the column headers in line 6, and the data starts in row 7.   We'll create a four-column table  to store the results.   In a SQL Server Management Studio (SSMS) window, create the following table:

use MyDatabase
go

create table stageFedRates (
 timePeriod date,
 jrxwtfb_nb dec(15,8),
 jrxwtfn_nb dec(15,8),
 jrxwtfo_nb dec(15,8)
 )
 ;

go

We have a couple of choices of programs to pull data from the web into an external file.  One option is wget, another is curl.   Curl is already installed on my Windows 10 computer and my Windows 2008 Server, so we'll use curl.   The syntax is pretty simple, just type curl and the URL enclosed in double quotes.  Don't forget the double quotes -- the URL is full of special characters that command console will try to interpret if not quoted.  The output goes to STDOUT by default.  Let's try it in a command shell window:

C:\> curl "https://www.federalreserve.gov/datadownload/Output.aspx?rel=H10&series=122e3bcb627e8e53f1bf72a1a09cfb81&lastObs=10&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package"

"Series Description","Nominal Broad Dollar Index ","Nominal Major Currencies Dollar Index ","Nominal Other Important Trading Partners Dollar Index "
"Unit:","Index:_1973_Mar_100","Index:_1973_Mar_100","Index:_1997_Jan_100"
"Multiplier:","1","1","1"
"Currency:","NA","NA","NA"
"Unique Identifier: ","H10/H10/JRXWTFB_N.B","H10/H10/JRXWTFN_N.B","H10/H10/JRXWTFO_N.B"
"Time Period","JRXWTFB_N.B","JRXWTFN_N.B","JRXWTFO_N.B"
2018-07-09,123.4946,89.6613,160.2890
2018-07-10,123.4637,89.7541,160.0923
2018-07-11,123.8276,89.8616,160.7776
2018-07-12,123.7232,90.1620,160.1318
2018-07-13,124.0710,90.2872,160.7554
2018-07-16,123.8317,90.0232,160.5674
2018-07-17,124.1553,90.4107,160.7806
2018-07-18,124.3573,90.5535,161.0480
2018-07-19,125.0530,90.9072,162.1562
2018-07-20,125.0484,90.9072,162.1456
c:\>

This is the same data we saw in the Excel spreadsheet above. Two useful options of the curl command are the --stderr option and the --output option.   Those two options specify files to store STDERR and the output file:

C:\>curl --stderr curlErrors.txt --output curlData.csv "https://www.federalreserve.gov/datadownload/Output.aspx?rel=H10&series=122e3bcb627e8e53f1bf72a1a09cfb81&lastObs=10&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package"

C:\>dir curl*.*
 Volume in drive C has no label.
 Volume Serial Number is 1836-25CF

 Directory of C:\

2018-07-28  03:38 PM               804 curlData.csv
2018-07-28  03:38 PM               399 curlErrors.txt
               2 File(s)          1,203 bytes
               0 Dir(s)  352,925,888,512 bytes free

C:\> type curlErrors.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   804    0   804    0     0    804      0 --:--:-- --:--:-- --:--:--  2233



Now that we can pull data from our source, let's look at loading into a SQL Server table.  From a new SSMS window,  use the bulk import command to load the table:

use MyDatabase
go

Bulk Insert stageFedRates 
   from  'C:\curlData.csv ' 
   with ( fieldterminator = ',',   
          rowTerminator = '\n',  
          firstRow = 7,
          maxErrors = 999999999 )


The fieldTerminator and rowTerminator options specify that the input is a CSV with rows delimited by new line characters and the columns by commas. We know there is some metadata in the download, so we specify firstRow=7 to start loading data. Finally, maxErrors specifies how many errors the import will tolerate before aborting the load. This will depend on the data; if the data is very clean, specify a small number of errors, but if not, specify a large number so all the data loads into the table for post-load cleanup.

In a SSMS window, select all the data and see what we imported into the SS table:

Data loaded into SQL Server


Now we're able to pull data from the web into a file and load that data into a table. Next, we'll create a stored procedure to automate this process. We've demonstrated all the commands we need to do this, but there's one more hurdle to leap: working within the context of the Windows file system permissions. The directories that we can read/write are not necessarily the directories that SQL Server service can read/write.

To work around this, we will query SQL Server's TMP environmental variable, and we will use that directory to write to and read from. The following stored procedure will do the trick. The comments will explain how it works along way. It's worth leaving the PRINT statements in the code for debugging; getting all the quotes and double-quotes correct can be quite tricky.

use MyDatabase
go

create procedure [dbo].[loadFedReserve] AS
begin
    -- Get SQL Server's tmp directory.    
    declare @destinationDirectory varchar(256);
    declare @cmd varchar(256) = 'echo %TMP%'; 

    -- Create a temporary table 
    create table #tmpResult (
 envVar varchar(256)
 )
    -- Store the results of executing "echo %TMP%" on the Windows server
    -- into the temporary table
    insert into #tmpResult exec xp_cmdshell @cmd 

    -- Now get the TMP directory from the table
    set @destinationDirectory = ( select top 1 envVar from #tmpResult);
 
    -- Build the command line, including the output and stderr file locations.
    -- We will reuse the @destination variable when we import the file to SQL Server.
    -- Be sure to use the double quotes around the URL  
    declare @curlCommand varchar(512) = 'curl ';
    declare @destination varchar(128) =  @destinationDirectory +'\curlData.csv ';
    declare @errors varchar(128) = @destinationDirectory + '\curlErrors.txt" ';
    set @curlCommand = @curlCommand + '--stderr "' + @errors
    set @curlCommand = @curlCommand + ' --output ' + @destination 
    set @curlCommand = @curlCommand + '"https://www.federalreserve.gov/datadownload/Output.aspx?' +  
       'rel=H10&series=122e3bcb627e8e53f1bf72a1a09cfb81&lastObs=10' + 
       '&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package"'
  
    -- Little bit of debugging, then execute the command
    print  @curlCommand; 
    exec xp_cmdshell @curlCommand
 
    -- The table is usually a staging table that we reuse, so delete any data in it.
    delete from stageFedRates; 

    -- Now build a sql statement with the bulk import command and the file locations
    -- that we determened before
    declare @sqlStmt nvarchar(256) = 'Bulk Insert stageFedRates from  ''' + 
  @destination + ''' with ( fieldterminator = '','', rowTerminator = ''\n'',  firstRow = 7,maxErrors = 999999999 )'
    
    -- A little more debugging, then execute the load.  Building the SQL command and 
    -- executing it with sp_executeSQL let's us change the imported file programmatically. 
    print @sqlStmt;
    exec sp_executeSQL @sqlStmt
 
    select count(*) RowsImported from fedRates
END

GO

Depending on how your SQL Server instance is configured, you may need to enable the use of xp_cmdshell. I can execute the following on the instance of SQL Server on my PC; the help of a DBA is probably required for an enterprise server.

exec sp_configure 'show advanced options', 1;  
go  
-- Refresh the configuration 
reconfigure ;
go 
  
-- Enable xp_cmdshell   
exec sp_configure 'xp_cmdshell', 1;  
go
  
-- Refresh the configuration 
reconfigure ;
go 
  

After compiling the store procedure, open a new SSMS window, execute the stored procedure, and select the results of the load:

exec loadFedReserve

select * 
from stageFedRates
 

There should be 10 rows, like below, and it will take SQL Server less than a second to pull the data and load the table:

Data returned from a SQL Select in SQL Server Management Studio
Using the curl command and a bit of code in a stored procedure, we can pull external data from the web and load it into SQL Server. After loading the data into the staging table,  we can manipulate the data as needed by our business requirements.   The load is fast enough to be useful for interactive processing with small amounts of data, or we can use the SQL Server Scheduler to batch load large amounts of data.



Saturday, April 7, 2018

Lubuntu 17.04 to 17.10 upgrade

Introduction


I'm a bit tardy upgrading my Linux Lubuntu 17.04 distribution.  I have two computers running 17.04, an old HP dv8210us with a single-core, 2GHz AMD processor, and a newer Asus netbook with a dual-core, 1GHz AMD processor.

Everything was working nicely until I tried installing Notepadqq, an open-source version of the popular Notepad++ for Windows computer.  For my next snafu, I tried installing libdvd to play DVDs in my HP.  Unfortunately,  Lubuntu 17.04 is no longer supported, and I couldn't load the pre-requisite packages for either libdvd or Notepaddqq.   Also, security updates are no longer available.

These are my notes and learnings as I worked thru the upgrade process from 17.04 to 17.10.     The notes are my collection of how-tos for the next time I do this.   The learnings are little bits of wisdom I picked up along the way.

First learning:   if you have a choice, install an LTS version of Ubuntu or Lubuntu.  Long Term Support (LTS)  versions are supported for 5 years (Ubuntu) or 3 years (Lubuntu).   This is the second time I've done an upgrade, first from 12.04 to 14.04, and now from 17.04 to 17.10.   The process is not as smooth as one would hope.  My systems are old, things will certainly work better or differently or at least more quickly on your own hardware.

Second learning:  take copious notes of everything you customized and how you customized it.   One problem area on the HP is the Broadcom wireless card.  If you know how to handle it, it's pretty simple.  But the first time thru was painful; following my notes in Google Keep made a messy process easy.

First Note:  Installation media


This is pretty simple,  I created a bootable USB for the netbook (no optical disk player) and a bootable DVD for the HP (no USB boot).  First download the iso image from Lubuntu.net.

To create the bootable USB, use the Startup Disk Creator (System Tools -> Startup Disk Creator). Select the iso image as the source,  a 2GB USB drive as the destination, then click the Make Startup Disk button.

To create the bootable DVD, use xfburn ( Sound and Video > xfburn).   The Burn Image button burns an ISO image to a disc.  A DVD  is required to hold the whole image; a CD is not quite big enough.

Third learning:  procrastination is not your friend.   I upgraded Ubuntu 12.04 to 14.04 using the software center over the network;    four months ago, I could have repeated the process and skipped creating bootable media.

Second Note:  Broadcomm wireless adapter


My old HP laptop has a Broadcomm wireless adapter.  The following notes will help you get this working.   If you don't have a Broadcomm wireless adapter, skip this section.

First, do the install while connected to the Internet with an Ethernet cable. If the system tries to connect to the internet using the existing driver, the system will hang.   You will need to download the correct driver from the internet.   So, plug in.

After installation is complete, remain connected to the Internet via the ethernet cable.  Start a terminal window, and execute the following four commands.
 
$ sudo apt-get update
$ sudo apt-get install firmware-b43-installer
$ sudo apt-get remove bcmwl-kernel-source
$ sudo reboot


Third note:  Consistency


Working on a Windows 10 PC all day at work, I'm accustomed to hitting Windows-L to quickly lock my screen.  I don't have a strong preference for Windows-L on my PC verses Ctrl-Alt-L on my Linux systems, but it's convenient when all the simple things work the same way.

To add this to Lubuntu, edit the lubuntu-rc.xml configuration file in your favorite text editor.  Search for the string "lxsession-default lock".  When you find the string, copy the entire <keybind> thru </keybind>, and paste it below the existing lock command.  Add a comment that the sequence Windows + L locks the screen, and change the "key=C-A-l" to "key=W-l":

  
  <!-- Lock the screen on Windows + l-->
  <keybind key="W-l">
    <action name="Execute">
      <command>lxsession-default lock</command>
    </action>
  </keybind>


Save the file, and from a terminal session refresh your openbox configuration:

openbox --reconfigure

Assuming you didn't get any errors, both Ctrl-Alt-L and Windows-L sequences will lock your screen.

Fourth note:   Two Monitors


I have a second monitor sitting to the left of my laptop screen.   If you know Lubuntu, you know  where this is going...  the left monitor is always the main monitor.  And if you're really "clever", you can swap the monitor positions but find that you don't have a taskbar with the laptop monitor alone.

I've googled this problem, and it's a common question with no satisfactory answer.  After a lot of trial and error, the best solution that I've found is to configure the menu on both monitors identically, and to use xrandr to set the monitors so the second monitor is to the left of the laptop screen.    This gives me a configuration that works whether the laptop has a second monitor or not, and my mouse moves left to go to the left monitor.  With the taskbar on both monitors, I will also have a taskbar on the laptop when only the laptop screen is in use.

Here's scoop:

First, right-click the task bar and open the panel settings.  Configure the panel settings to your personal preferences and close the window.

Next, open a terminal window, and change your working directory to .config/lxpanel/Lubuntu/panels.   Edit the panel file in the panels directory using a text editor like vi or leafpad.  In the global section, add one line

monitor=0

and save the file.   Now change the monitor=0 line to monitor=1, and using a Save as... dialog, save the panel as panel1.

Execute lxpanelctl restart, and you will have taskbars on both of your monitors.

Fifth Note:  Network and Volume Indicators


After installing 17.10, I noticed problems with the network and volume controls on the task bar:  The network menu collapsed so quickly it was almost unusable, and the volume control didn't work at all. After a lot of googling,  I found that this is only a problem if the autohide option on the taskbar is selected.  I like the autohide feature, so I worked around this problem in two ways.   I rarely use the network menu; if I need it, I turn off the autohide feature while I use the network menu.

I use the volume control frequently; so turning off autohide every time I want to mute an advertisement is not ideal.   Instead, I disabled the Sound Menu on the indicator menu and added the Volume Control plug in to the panel menu:
  • Right click on the task bar, and select Panel Settings
  • Click on the Panel Applets tab, select Indicator applets, and click the Preferences button
  • Uncheck the Sound Menu and close the Indicator applets menu
  • Back on the Panel Preferences menu, click the Add button
  • On the Add plugin to panel menu, scroll down the list, find the Volume Control, highlight it, and click the Add button
  • The Volume Control is added to the right of the power control.  Use the Up and Down buttons to move the control where you like.
  • Close the panel preferences

Sixth Note:  More on Network Interfaces


I have two wireless network interfaces with my Asus netbook.  The built-in Qualcomm Atheros AR9485 worked fine out of the box.   In addition, I have a Realtek RTL8192cu USB dongle.   When booting into Windows 10, I found the Realtek interface is much faster than the Qualcomm interface.   In Lubuntu, the Realtek interface would stop working every two or three minutes until I disabled/enabled the interface. Then it would work again for another two or three minutes.

Googling the issue turned up several different answers to the problem; the references below all agree that it's a driver issue.  In short, the driver in kernel 4.4 is supposed to work well, but Lubuntu 17.10 is using kernel 4.13, and the driver for the Realtek interface is buggy.  Here's how to fix it.

sudo apt-get install linux-headers-$(uname -r) build-essential dkms git
git clone https://github.com/pvaret/rtl8192cu-fixes.git
sudo dkms add ./rtl8192cu-fixes
# NB:  The step above reports a version.   
#      As of 2018-04-12, the version is 1.11, not 1.10.
#      Use the correct version in the next step.
sudo dkms install 8192cu/1.11
sudo depmod -a
sudo cp ./rtl8192cu-fixes/blacklist-native-rtl8192.conf /etc/modprobe.d/
sudo cp ./rtl8192cu-fixes/8192cu-disable-power-management.conf /etc/modprobe.d/
reboot


Seventh Note:  Printers


I use two printers at home, a old Lexmark Optra S 1255, and a wireless Epson NX420.  Configuring the Lexmark is easy; it's a postscript printer, and Linux has always supported postscript printers well.   Open System Tools -> Printers -> Add.  I have a network card in the printer, so I expand the Network Printer options and choose AppSocket/HP JetDirect.  Enter the IP of the printer, port 9100 is already filled in, and click Forward.  Select the manufacturer from the list, then the model,  describe the printer, click Apply and print the test page.

The default driver for the Epson NX420 does not work well at all; the colors do not align properly, but instead look like shadows.   This is another problem that's a pain to fix the first time, but it's not too bad the second time.  The NX420 needs a driver from http://www.openprinting.org/printer/Epson/Epson-NX420_Series.  Choose the 64-bit deb driver, download it, and then open the downloaded file.   Lubuntu will open the driver in the Software Center.   Click the Install button to install the driver.

Now add the printer ( System Tools -> Printers -> Add).   Expand the network printers, select the NX420, and click the Forward button.  The system searches for print drivers for a minute or so, then displays the Describe Printer dialog.  Click Apply, and print a test page to verify that the driver works properly.  The test page should identify the driver as EPSON.PPD, and the test page should look perfect.   If the print driver is STP01629.PPD and the images at the top of the test page don't align, then you have the bad print driver.

With these notes, I'm ready for the next upgrade.    With Lubuntu installed on my 12-year old laptop and my under-powered netbook,  I have two pretty responsive computers.   The laptop is perfect for my desk:  if the cat walks across it, who cares? It's a twelve year old computer.  And the one and half pound netbook is great for traveling.


References





Tuesday, January 23, 2018

Snipping in Lubuntu, the sequel

In my last post, I added screen snipping to my Lubuntu system using scrot and mtpaint.   The result is similar to using the Snipping Tool in Windows 7, where the resulting clipped image is left in the Snipping Tool viewer.    There are two drawbacks to the approach:  first, leaving the clipped image in mtpaint is not the quickest thing to do for simple copy and paste applications, and second, not every program will accept the resulting image.

In this post, I'll demonstrate adding a new meta-key combination, Windows-Shift-S to select and copy part of a screen, format the data so any program can use it, and copy the image to the system clipboard.  The end result will be the same meta-key that Windows 10 uses to capture screens, and the result will be ready to paste -- no intermediate step leaving the result in mtpaint.

Capturing the image in a format that was usable by all my programs turned out to be trickier than I thought.   At first, I thought I could send the output of scrot to xclip.   This works for LibreOffice Writer and Gmail, but it didn't work for Blogger.  Hence,it was time to take a deeper dive into this process.  The 'net being the 'net, there's often someone who's run into the same problem.   In this case, artfulrobot (their handle) on askubuntu.com provided a solution that I'm going to work with here.

In this post, I'll discuss using scrot, base64, and xclip to add Snipping Tool functionality to Lubuntu.  We'll start by making sure we have all three programs installed, then we'll test invoking them from the command line, and finally we'll map them to meta-keys in the Openbox window manager.

First, make sure these the tools are installed on your system.   Base64 should be installedd on Lubuntu, but you may need to install scrot and xclip yourself.  Open a terminal window, and try executing xclip and scrot; if they load, everything is good.   If you get a message like:

The program 'xclip' is currently not installed. You can install it by typing: 
sudo apt install xclip 


If you get that message, go ahead and install the tool using apt:

sudo apt install xclip


Now you should have all three tools installed.  There are a lot of moving parts, so let's start with the solution and work backwards.  If you haven't read the previous post, I suggest you browse it now.

First, edit the openbox configuration file using vi or your favorite text editor:

tom@dv8000:~$ cd .config/openbox/ 
tom@dv8000:~/.config/openbox$ ls -l 
total 36 
-rw-r--r-- 1 tom tom 33058 Jan 23 14:06 lubuntu-rc.xml 

tom@dv8000:~/.config/openbox$ cp lubuntu-rc.xml lubuntu-rc.xml.safe
tom@dv8000:~/.config/openbox$ vi lubuntu-rc.xml 


 In our previous post, we added the Windows-s meta-key to copy a screen shot to mtpaint in lines 384 through 389.  The command to do this on line 387 was pretty simple, and we included it right in the configuration file.

The command we need to execute this time is a bit too complicated to include directly in the openbox configuration file, so we'll wrap it in a simple shell script on line 394.  The easiest place to put the shell script is in your home directory until you're satisfied with it.  In my case, that's /home/tom; I'll name the shell script ScrotIt, a meshing of SnipIt and scrot.  Using your editor, add lines 390 through 397, substituting your home directory for /home/tom:


    384     <!--  Windows-s key to select screen and copy to mtpaint -->
    385     <keybind key="W-s">
    386       <action name="Execute">
    387         <command>  scrot -s -e 'mtpaint $f ' /tmp/ScrotSave.png </command>
    388       </action>
    389     </keybind>
    390     <!-- Windows-Shift-s to select screen and copy to xclip -->
    391     <keybind key="W-S-s">
    392       <action name="Execute">
    393         <command>
    394             /home/tom/ScrotIt
    395        </command>
    396      </action>
    397     </keybind>



After you added the new keybinding, save the file and refresh openbox with your changes:

openbox --reconfigure

Next, edit the script ScrotIt using your favorite editor, and add the following lines:

      1 #!/bin/bash
      2 scrot -s /tmp/scrotSave.png && \ 
      3    echo "<img src='data:image/png;base64,"\ 
      4         $(base64 -w0 /tmp/scrotSave.png)\
      5         "' />" | 
      6    xclip -selection clipboard -t text/html 


Let's walk through the code and see what's happening:


  • Line 2 runs the scrot command, saving the output in /tmp/scrotSave
  • Lines 3 thru 5 build an ASCII string with the image
  • Line 3 starts an <img> tag, and the source is a data URI.   The MIME type is image/png, and it will be base64-encoded.
  • Line 4 runs the base64 command, encoding the image data as an ASCII character string.  The bash $( ) construction executes base64 and returns the output as a string to the script running the command
  • Line 5 closes the <img> tag we started in line 3.  The shell concatenates the string in line 3, the output of base64 in line 4, and the closing tag string in line 5 into one long string.
  • Finally, on line 5 the output of the echo command that started on line 3 is piped to xclip on line 6.  xclip takes the output and leaves it in the system clipboard, ready for pasting.

The last step is to make the script executable and test it.  After typing in the ScrotIt command on the second line and pressing enter, ScrotIt will wait until you select an area of your screen.

 

tom@dv8000:~$ chmod +x ScrotIt 
tom@dv8000:~$ ./ScrotIt 


Select an area of your screen and paste it into an application that supports images, like your word processor or mail program (but not a text editor like vi). I copied and pasted the following image into blogger:

Linux terminal screenshot showing Scrotit running

If all went well, now try it with the Windows-Shift-S meta-key combination that you defined for openbox.   After selecting an area of your screen with your mouse, you should be able to paste it into your document.

Using these tools, you can select, copy, and paste from your screen into documents as easily with Linux as you can with Windows 10.



References:

In addition to the Unix man pages for scrot, base64, and xclip, I found the following web pages to be useful.

https://wiki.archlinux.org/index.php/openbox#Keybinds

https://askubuntu.com/questions/759651/how-to-copy-an-image-to-the-clipboard-from-a-file-using-command-line

https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs

Saturday, January 13, 2018

Snipping in Lubuntu

One feature that I find very useful in Windows 10 is the Snipping Tool.   The Snipping Tool is mapped to Win-Shift-s key combination.   After pressing that meta-key combination, the screen grays out, and you can select an area of the screen to copy to the system clipboard.    This is really useful for splatting images quickly into an email (or other document) with a minimum of effort.   Print Screen and Alt-Print Screen were useful for grabbing images of a whole screen or of a whole window.   But many times we only want a small portion of the screen, and the Snipping Tool is a useful way to grab exactly what you need.

In my last post, my old laptop got a second lease on life with a Lubuntu installation and an upgrade to 2GB of memory.  I am spoiled  by the Snipping Tool on my Windows 10 computers, so I'm looking for a way to get that functionality with Lubuntu.

As with many things in life, "da google" is your friend.   Googling "screen capture" or "screen image" returns lots of hits showing screen images from Lubuntu, but not a lot of information describing how to grab that screen image.    Eventually, I found some information about a screen capture utility called scrot (short for SCReen shOT).  In this post, I'll discuss using scrot and mtpaint to add Snipping Tool functionality to Lubuntu.  Mtpaint is installed as part of the Lubuntu distribution, but you may need to install scrot  yourself. Finally we'll map them to meta-keys in the Openbox window manager.

First, make sure scrot and mtpaint are installed on your system.     Open a terminal window, and try executing scrot -v.  If the tool loads and displays a version number, everything is good.   If it doesn't load, you'll see a message like:

The program 'scrot' is currently not installed. You can install it by typing: 
sudo apt install scrot 


If you get that message, go ahead and install the tool using apt:

sudo apt install scrot


Now you should have both tools installed.  I'll start by running them from the command line.  First, try
scrot --select

You'll notice that your terminal window pauses, waiting for you do do something.  With your mouse, click and drag over any section of your screen, creating a rectangle.  Release the mouse button, then in your terminal window type
ls -l *png

and your output will look something like the following.   Scrot creates files with the datetime and images size in the filename:

tom@dv8000:~$ ls -l *png
-rw-rw-r-- 1 tom tom 131103 Jan 13 11:48 2018-01-13-114835_1440x875_scrot.png
-rw-rw-r-- 1 tom tom    510 Jan 13 11:48 2018-01-13-114856_184x172_scrot.png
tom@dv8000:~$


Our next step is to get the the output into mtpaint.  Scrot has a couple of options to facilitate sending the captured image to another program.   The -exec option tells scrot to run another program, and the $f will pass the name of the saved file.   Try the next command; scrot will save the output in /tmp/ScrotSave.png and then execute mtpaint with that filename as the argument.

scrot --select --exec 'mtpaint $f ' /tmp/ScrotSave.png

Now we have all the facilities of mtpaint at our disposal to edit the image.  Using mtpaint, we can save that image back to disk, or just copy it to the system clipboard  using Ctrl-C, then Edit -> Export Clipboard to System from the drop-down menu.

Finally, we'll map this command to a meta-key combination in Lubuntu's openbox window manager.   From your home directory, navigate to the .config directory, then to the openbox directory, and look for the XML configuration file.   In my system, it's lbubuntu-rc.xml:

tom@dv8000:~$ cd .config 
tom@dv8000:~/.config$ cd openbox  
tom@dv8000:~/.config/openbox$ ls -l 
total 36 
-rw-r--r-- 1 tom tom 33422 Jan 13 15:10 lubuntu-rc.xml 
tom@dv8000:~/.config/openbox$ 


There are lots of things that we can configure here, but we're only interested in the keybindings. Before you begin, copy the xml file to a safe place! If you make an error, you don't want to leave the configuration file a mess, and you can restore it using your saved copy.  After creating a backup copy, edit lubuntu-rc.xml with your favorite text editor (leafpad and vi are both installed on Lubuntu, don't use a document editor like Abiword, OpenOffice, or LibreOffice). 

Find the line where the Lubuntu specific keybinding start, and add the XML starting with the <keybind> tag through the closing </keybind> tag:

    382     <!--  Lubuntu specific : Keybindings -->
    383     <!--  Lubuntu specific : Keybindings -->
    384     <!--  Windows-s key to select screen and copy to mtpaint -->
    385     <keybind key="W-s">
    386       <action name="Execute">
    387         <command>  scrot -s -e 'mtpaint $f ' /tmp/ScrotSave.png 
    388      </action>
    389     </keybind>
 

Let's look at what we've done:   Line 384 is a comment describing our change. In line 385, we start a new keybinding, capturing the meta-character combination Windows and the "s" key.  The action will be to Execute the command within the command tags on line 387, which is the scrot command that we executed in the terminal prompt above.

Next, save the file, and refresh the window manager with the new configuration information:

openbox --reconfigure

If all went well, openbox will not display any output.   If there is an error, openbox pops up a message describing the error and its location in the XML file.   After openbox has reloaded the XML file, test your keymap by pressing the Windows and S, then select part of your screen using the mouse.   After you release the mouse key, mtpaint will open with a copy of the image you selected. 

Now you have a quick and easy way to grab a selected part of your screen and manipulate it for further use.  Save the image as a file, or copy it to the system clipboard in mtpaint.




References:

In addition to the Unix man pages for scrot and mtpaint,  the following web pages should be useful for readers desiring a deeper look into the keybinding and mtpaint.

https://wiki.archlinux.org/index.php/openbox#Keybinds

http://mtpaint.sourceforge.net/handbook/en_GB/chap_04.html#SEC6