formatting notes for blogger

Vim

Disable search wrapping (temporarily for vim script)

https://groups.yahoo.com/neo/groups/vim/conversations/topics/33407

Recursive macros

http://vim.wikia.com/wiki/Record_a_recursive_macro

Try/catch blocks for vim script?

http://superuser.com/questions/319048/vim-continue-macro-after-error-in-submacro

Search patterns

http://vim.wikia.com/wiki/Search_across_multiple_lines

Regexp

Reference manual

http://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap09.html

Using "and" blocks for regular expressions

http://stackoverflow.com/questions/469913/regular-expressions-is-there-an-and-operator

Emmet

Reference highlighted material in an abbreviation (like creating anchor tags for hyperlinks).

http://docs.emmet.io/actions/wrap-with-abbreviation/

  a[href=$#]{$#}

Additional references:

• https://code.google.com/p/zen-coding/wiki/ZenHTMLSelectorsEn

Cheat sheet

http://docs.emmet.io/cheat-sheet/

Java: Https clients, headers and cookies

Sample HTTPS client

Processing the input stream is more reliable than using the #getContent() method.

http://alvinalexander.com/blog/post/java/simple-https-example

  package foo;
   
  import java.net.URL;
  import java.io.*;
  import javax.net.ssl.HttpsURLConnection;
   
  public class JavaHttpsExample
  {
    public static void main(String[] args)
    throws Exception
    {
      String httpsURL = "https://your.https.url.here/";
      URL myurl = new URL(httpsURL);
      HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();
      InputStream ins = con.getInputStream();
      InputStreamReader isr = new InputStreamReader(ins);
      BufferedReader in = new BufferedReader(isr);
   
      String inputLine;
   
      while ((inputLine = in.readLine()) != null)
      {
        System.out.println(inputLine);
      }
   
      in.close();
    }
  }

Additional references:

• http://www.mkyong.com/java/java-https-client-httpsurlconnection-example/
• http://www.javaworld.com/article/2077600/learn-java/java-tip-96--use-https-in-your-java-client-code.html

Url parameters (GET and POST)

Depending on the method for the request, there are a couple different ways that url parameters can be configured. GET requests will include the parameter names and values in the url itself. POST will include the parameter names and values in a request header.

GET

  String http_url = "http://127.0.0.1:3771/greeting/test";

  String charset = "UTF-8";  // Or in Java 7 and later, use the constant: java.nio.charset.StandardCharsets.UTF_8.name()
  String param1 = "value1";
  String param2 = "value2";

  URL url;
  try {

    String query = String.format("param1=%s¶m2=%s",
           URLEncoder.encode(param1, charset),
           URLEncoder.encode(param2, charset));

    url = new URL(http_url + "?" + query);
    HttpURLConnection con = (HttpURLConnection)url.openConnection();
    con.setRequestProperty("Accept-Charset", charset);
    ...

  } catch (Exception e) {
		e.printStackTrace();
  }

POST

  http_url = "http://127.0.0.1:3771/greeting/shake.json";

  String charset = "UTF-8";  // Or in Java 7 and later, use the constant: java.nio.charset.StandardCharsets.UTF_8.name()
  String param1 = "value1";
  String param2 = "value2";

  URL url;
  try {
    url = new URL(http_url);

    con = (HttpURLConnection)url.openConnection();
    con.setRequestMethod("POST");
    con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);
    con.setDoOutput(true);

    String query = String.format("param1=%s¶m2=%s",
           URLEncoder.encode(param1, charset),
           URLEncoder.encode(param2, charset));

    try (OutputStream output = con.getOutputStream()) {
        output.write(query.getBytes(charset));
    }
    ...
    
  } catch (Exception e) {
		e.printStackTrace();
  }

Additional references:

• http://stackoverflow.com/questions/2793150/using-java-net-urlconnection-to-fire-and-handle-http-requests

The nature of cookies

Cookies are only set by the server; any modifications to the cookies on the client side are ignored (at least when using java.net's CookieManager class). Cookies are stored on the client side and the client may read them.

http://curl.haxx.se/rfc/cookie_spec.html

  HttpCookie cookie = new HttpCookie("blah", "bleh");

  url = new URL(http_url); 
  cookieJar.add(url.toURI(), cookie);
  HttpURLConnection con = (HttpURLConnection)url.openConnection();

Accessing Cookies, both retrieval and assignment.

• http://docs.oracle.com/javase/tutorial/deployment/doingMoreWithRIA/accessingCookies.html
• http://www.hccp.org/java-net-cookie-how-to.html

HTTP response code: 411

When sending a POST request to a server, there must be a non-null body provided, even if it is an empty string or an exception will be thrown. The behavior for other libraries may differ, but for the java.net package, this is the case.

  java.io.IOException: Server returned HTTP response code: 411 for URL: http://127.0.0.1:3771/greeting/shake.json

You will need to write to the output stream to overcome this problem. Doing so automatically creates the "Content-Length" request header.

  con.setRequestMethod("POST");

  con.setDoOutput(true);
  BufferedWriter bf = new BufferedWriter(new OutputStreamWriter(con.getOutputStream()));
  bf.write("");
  bf.flush();

Additional references:

• http://stackoverflow.com/questions/2651379/webrickhttpstatuslengthrequired-error-when-accessing-create-method-in-contro
• http://stackoverflow.com/questions/15619562/getting-411-length-required-after-a-put-request-from-http-client
• http://docs.oracle.com/javase/7/docs/api/java/net/URLConnection.html#setDoInput%28boolean%29
• http://stackoverflow.com/questions/9767952/how-to-add-parameters-to-httpurlconnection-using-post

Setting request headers and their respective values

http://stackoverflow.com/questions/6469540/setting-custom-http-request-headers-in-an-url-object-doesnt-work

  URL url = new URL("http://myipcam/snapshot.jpg");
  URLConnection uc = url.openConnection();
  uc.setRequestProperty("Authorization", 
    "Basic " + new String(Base64.encode("user:pass".getBytes())));

  // outputs "null"
  System.out.println(uc.getRequestProperty("Authorization"));

http://stackoverflow.com/questions/12731211/pass-cookies-from-httpurlconnection-java-net-cookiemanager-to-webview-android

  webCookieManager.setAcceptCookie(true);

Print out all header keys and their respective values.

http://www.mkyong.com/java/how-to-get-http-response-header-in-java/

	//get all headers
	Map> map = conn.getHeaderFields();
	for (Map.Entry> entry : map.entrySet()) {
		System.out.println("Key : " + entry.getKey() + 
                 " ,Value : " + entry.getValue());
	}
 
	//get header by 'key'
	String server = conn.getHeaderField("Server");

Ruby/Rails: access cookies

http://api.rubyonrails.org/classes/ActionDispatch/Cookies.html

  cookies[:user_name] = "david"
  cookies.signed[:user_id] = current_user.id

Ruby/Rails: access headers

http://stackoverflow.com/questions/19972313/accessing-custom-header-variables-in-ruby-on-rails

  request.headers['custom_header']

Now custom variables are always prepended with HTTP_ ... except for CGI variables Harsh Gupta

The Death of a Hard Drive

Every had that moment you walk into the office and see a white/gray screen with a folder and a question mark on your iMac? Or better, how about a blank white/gray screen? Oh, yes. Your hard drive isn't feeling too good.

At this point you grab your laptop (that is working) and you start doing searches to try and see how to rescue your computer. In my case, I had a pretty good hunch it wasn't the processor chip.

1. the screen came on

If the screen doesn't light up, especially if the BIOS screen doesn't come up (for Windows), you might be looking at a bad CPU. Sure, there are several possibilities at this point, but the next logical step I would take is to check the health of the hard drive.

Check the health of the disk

1. get an installation disk or thumb drive that you can boot off of; if using the installation disk, insert the disk while the computer is on
2. turn off the computer by holding the power button until it powers off
3. turn on the computer, wait for the initial chime and hold the Option key down until you are prompted to select which device to boot from (there are other key combinations as well); boot from the installation disk or appropriate usb device

Once you are booted external to the potentially fouled up hard drive, you want to use Disk Utility to check on the internal hard drive's health. With the installation disk, you may feel startled by the new installation screens. Don't worry, there are instructions on the second screen you can follow to load Disk Utility and not install the OS.
Once Disk Utility has loaded, if you can see that the internal hard drive did indeed mount (and is not grayed out), you can run verify to see if there are problems with the hard drive.
If there are, and they are not simply permission issues, you have some choices to make.

OK, it's broken; what do I do now?

1. Use the Repair option in the Disk Utility tool (I do not recommend this; I'll explain why later)
2. plug in an external USB drive, close Disk Utility and open Terminal, using command line calls to copy data from the damaged hard drive to the external USB drive
3. use a data recovery tool (like Data Rescue) to first clone the dying hard drive and then pull data from the clone

I would strongly recommend option 3 as you really don't know how much longer the hard drive will last.

My observations (for what they are worth)

Here are some things I have discovered with my last two data recovery sessions (hard drives on a 10 year old MacBook and iMac recently died).

• the hard drive will need to be replaced; sure, you might be able to put a band-aid on the hard drive in question, but it is compromised and you will be left rescuing data from a dying hard drive again and soon.
• repairing the drive involves changes to the hard drive that may end up leaving the stability of the system worse than it was before; if your computer is responding abnormally slow, this is another indication that your hard drive may be failing
• performing a clean or defrag (even on a Windows VM hosted on a Mac) could have the same results as using Disk Utility's Repair
• even with the data recovery tool, each time the disk was scanned, there were more reports of slow reads. This is why I strongly suggest doing a clone before doing anything else if you even suspect the drive might be dying. Then you use the data recovery to pull files from the cloned drive.

In my particular case, the installation disk method worked until I attempted to repair the disk. The results of doing so claimed that all was well, but when I defragged my Windows VM and rebooted the computer, I got the white screen again. Loading Disk Utility via the installation disk, this time the internal hard drive would not mount. And 5 more attempts did not bring forth any fruit.
At this point, I decided to take the hard drive out and use a USB hard drive connection tool meant for data rescue efforts. There seem to be plenty of YouTube videos that provide instruction on how to open up an iMac to remove the hard drive, so I checked them out. I ended up using a toilet plunger (it was clean) to suck the screen cover off since I didn't have the fancy tool. If you are opening up your iMac, you will also need the correct size star driver (the small, precision kind).
Once the hard drive is out, you will need a USB SATA/IDE converter. Most of these converters will have a power source as well, which you will need. I plugged my hard drive into the USB converter, plugged in the USB cable and fired up my data recovery software (e.g., Data Rescue seems to be a popular one), and performed a quick scan. It didn't take me long to realize I needed to do a clone first (which requires a dedicated external drive) as the reported list of slow reads got longer and longer. After performing a clone of the dying hard drive (which took 5 hours during the night and 365 slow read detections), I was finally able to rescue my data.

Prevention is the best medicine

Of course, the best course of action is to be backing up your important data on a frequent basis and there are many tools that can help. Hard drives will eventually fail -- it's just a fact. Rather than wait until a drive is failing, consider

• using a tool like Carbon Copy Cloner and an external hard drive for each computer hard drive
• use a wireless device (like Time Capsule) that all your computers can connect to and perform regular backups
• use an online service like BackBlaze and your WiFi to backup your computer drives

I'm seriously considering using the online service option.
One popular option is BackBlaze which costs about $40 per computer a year. When you factor in the cost of additional external hard drives, the purchase of data rescue software and the long hours, agonizing over a dying hard drive, it might very well be worth it.

using grouper to synchronize LDAP with Active Directory

My current assignment is to use Grouper to synchronize group memberships between an LDAP and an Active Directory (AD).  Even though AD is very much like an LDAP, it is not.  What's more we don't have control over making changes to AD because we are using the Microsoft Cloud.

We want to be able to use a single tool to handle group management and make those groups available via LDAP and AD.  AD does not support dynamic groups to the degree that we need, so we plan on including DN values explicitly for each group.  Our LDAP does support dynamic groups, which we are currently using.

Some applications connect to LDAP while others must connect to AD. We need a solution that handles the following:

1. AD groups are provisioned with explicit lists of DN values
2. LDAP DN values differ slightly from AD DN values and will require a transformation from “uid=dvezzani,...” to “cn=dvezzani,...”

In order to achieve this goal, we plan on primarily using the grouper-loader to pull in DN values from LDAP and psp to provision groups with the transformed DN values.

LDAP and AD subjects are being populated by separate means, but they both contain the same logical set of subjects. Is this the right approach to accomplish our goals?

JMeter on my Mac, Permission denied

In response to why JMeter didn't start up for me on my Mac. There are differences between the zip and tgz archive versions. In regards to the zip version...

That is intended for Windows systems, as the zip format does not allow one to set the executable bit for files.

http://jmeter.512774.n5.nabble.com/Jmeter-2-3-2-not-working-in-Linux-td535153.html - sebb-2-2

I downloaded the tgz file and JMeter works like a charm now.

Solar Power - hype or help?

So what's all the hype with solar power? You may hear things like

• you won't every have to pay an electricity bill again
• PG&E will cut you monthly checks for your excess power
• you will be a hero because you'll be green

"Solar panels in Ogiinuur" by Chinneeb - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons.

I'm certainly not going to try and fill you in on everything that we've been learning. We just spent 4 hours with a wonderful solar power representative. Here are some things to watch out for.

• small print indicating the company can simply walk away from the agreement at anytime, un-obligated, leaving you to manage the array of solar panels on your roof on your own
• who will be monitoring the equipment, handling the maintenance of warrantied panels, replacing them at the end of 30 years, replacing the power inverter? You? Why not? You’ve got nothing better to do with your time than try to chase down licensed/bonded contractors that you trust to service your solar panels, right?
• will the company still be around through the next 20 years? If not, will they leave you on your own to deal with the solar panels on your roof?
• if you move, do you want to have the option of transferring your solar panels to your new house?

This list is far from exhaustive, but hopefully it gets you thinking about some important questions to ask if/when you meet with sales representatives. This is a big investment and, if you're like us, you don't have money to throw around. Make sure you do your home work and shop around.

There are some other flags that I find useful.

• is the sales representative salary paid or strictly by commission? Knowing that the sales rep gets paid whether I commit or not takes some of the pressure off the meeting and allows more open and candid conversation
• how many backers does the solar company have? Do you recognize the company names? What kind of capital investment does it have? A solar power company with $2 billion dollars behind it stands more chance of being around when your 20 contract is up.

One thing that caused my wife and I to cringe as we considered a contract with a solar power company was the bottom line of how much it would cost. But when you compare it to the best scenario cost of the current electricity expenditures, we end up saving $20 thousand dollars. Perspective is everything. Basically, the way the business model works is

• you stop paying PG&E and start paying your monthly electricity to the solar power company.
• the solar power company should review through your power usage over the past year and determine the average kilowatt hours you consume each month.
• then a solar panel solution is determined that would provide the needed kilowatt hours
• solar power is produced and a power inverter moves the power to PG&E's power grid and you continue powering your appliances from PG&E's power grid. It's not a perfect comparison, but think of the power grid as a giant battery that's being recharged, in part, by the solar panels on your roof.
• if you use more kilowatt hours than your solar panels produce, you will simply pay PG&E for it at the end of the year.
• if you use fewer kilowatt hours than your solar panels produce, that will adjust the bottom line of what you owe to PG&E (or the check that PG&E will write out to you!) at the end of the year.

There are many other details to consider, but you have the basic idea.

Don't get too excited of the prospect of simply putting extra panels on your roof and start your own power making business.

• the exchange rate is abysmal; for every excess 15 kilowatt hours, PG&E may pay you for 3 kilowatt hours
• there are probably some laws in place which discourage it

Still, just getting a check instead of a bill from PG&E should make anybody's day.
At the end of the day, most of us simply want to find ways to save a little money. So far, I am discovering that solar power would be a great way for our family to cut our current bill by more than half. I don't even need to pay anything initially for the solar panels, their installation or maintenance.

For those who have 150 thousand laying around, you will save even more! How does 20 years without an electricity bill sound? Yeah; I thought that too! Perhaps you can purchase solar panels for less. Make sure you understand what you get for your price. Does that purchase include service for the panels through the entire 20 years, even if the company goes out of business? Make sure you do your homework.

So what's all the hype with solar power? It's still relatively new, but the kinks are getting worked out and we can all start enjoying a more consistent and lower power bill thanks to solar technology for zero down. The devil is in the details and not all solar companies are the same. Meet with several companies and ask questions, like the ones in this post; compare and find which company works best for you and your situation.

We can all use a little more money in our budget. How about... now?

'or'-ing terms with bash and grep

Needing to search for lines in a group of files having either of the supplied terms.

grep -e foo -e bar *.txt

With a more complete context:

  for file in $(ls aaa.txt bbb.txt)
  do
    echo ""
    echo $file
    echo "======================="
    cat $file | grep -e "^foo" -e "^bar"
  done

http://unix.stackexchange.com/questions/37313/how-do-i-grep-for-multiple-patterns

removing app from launchpad

Accidentally dragged an app onto Launchpad instead of the Applications directory.  Needed to remove it and found this:

http://osxdaily.com/2012/01/05/remove-apps-from-launchpad/

... but I had problems running the snippet as it was.  I ended up running something like this:

sqlite3 "/Users/john/Library/Application Support/Dock/1426ED5E-E2B5-43CF-9616-698ED687577D.db" "DELETE from apps WHERE title='SomeApp';" && killall Dock

Apparently, I had to provide a full path to the sqlite3 database.  Running 'ls' pulled up not one, but two entries.  The command to sqlite3 only takes a single database file.

quick and dirty SSL client using Ruby and Savon

I needed to test my Apache CXF web service using a particular SOAP request stored in a plain xml file. Since I am familiar with Ruby (and I'm always looking for excuses to use it), I used Savon as a tool to start from. I added the ability to provide a file whose contents is the desired SOAP request. I also needed to send my request via HTTPS and Savon was already able to accommodate (thanks to it's own use of HTTPI).

To use my forked version of savon:

gem 'savon', :git => 'https://github.com/dcvezzani/savon.git'

Once a Savon client has been created and initialized with the necessary WSDL and SSL configuration, the call is simple:

call_options = {body_content: body_content}
response = client.call(:soap_operation, call_options)

See the sample client for details.

Need to extract private and public keys out of a Java Keystore (JKS)? See my other post.

extracting private, public and trusted keys from a Java Keystore (JKS)

Provide your own particular values.

• STORE_PASS: the keystore password
• STORE_LOCATION: the absolute path for your keystore location
• TARGET_ALIAS: the alias name for the targeted key in the keystore
• WSDL_URI: the uri (or url) for the WSDL associated with the service that will be called
• TRUSTED_ALIASES: list of alias values to be extracted out of JKS file and put into PEM formatted truststore file

  export STORE_PASS=
  export STORE_LOCATION=/Users/joe/...
  export TARGET_ALIAS=
  export WSDL_URI=https://...
  export TRUSTED_ALIASES=1("one" "two" "three")

Verify your environment variables

keytool -list -v -storepass ${STORE_PASS} -keystore ${STORE_LOCATION} | less

filename=$(basename "$STORE_LOCATION")
suffix="${filename##*.}"
prefix="${filename%.*}"
alias=${TARGET_ALIAS}

Export the public key

keytool -export -rfc -keystore ${STORE_LOCATION} -storepass ${STORE_PASS} -alias ${alias} -file ${prefix}.pub

Export the private key. This is necessary because some tools don't know how to use a Java keystore.

keytool -importkeystore -srckeystore ${STORE_LOCATION} -destkeystore ${prefix}.p12 -srcstoretype JKS -deststoretype PKCS12 -srcstorepass ${STORE_PASS} -deststorepass ${STORE_PASS} -srcalias ${alias} -destalias ${alias} -srckeypass ${STORE_PASS} -destkeypass ${STORE_PASS} -noprompt

openssl pkcs12 -in ${prefix}.p12 -out ${prefix}.key -passin pass:${STORE_PASS} -passout pass:${STORE_PASS}

openssl x509 -in ${prefix}.pub -text -noout
openssl rsa -in ${prefix}.key -check

Verify that private/public key pair were extracted correctly

export WGET_CONFIG="--secure-protocol TLSv1 --no-check-certificate --certificate ${prefix}.pub --certificate-type PEM --private-key ${prefix}.key --private-key-type PEM"

wget ${WGET_CONFIG} ${WSDL_URI}

Remove password on the private key (dangerous!). Make sure the file permissions, at least, are locked down.

openssl pkcs12 -in ${prefix}.p12 -out ${prefix}.pem

openssl rsa -in ${prefix}.key -out ${prefix}-no-password.key
chmod 400 ${prefix}-no-password.key

Create a truststore file in pem format.

echo "" > ${prefix}-truststore.pem
for trusted_alias in "${TRUSTED_ALIASES[@]}"
do
keytool -export -alias ${trusted_alias} -file trust-${trusted_alias}.crt -keystore ${STORE_LOCATION} -storepass ${STORE_PASS}
openssl x509 -inform der -in trust-${trusted_alias}.crt -out trust-${trusted_alias}.pem
cat trust-${trusted_alias}.pem >> ${prefix}-truststore.pem
done

Does the formatting of a tnsnames.ora file really make a difference?

I recently set up Oracle on my Mac so I could develop an application that uses Oracle for the database tier. I use the option of running Oracle server on a VM and configuring it to be available to my Mac using port-forwarding.

• http://dimitrisli.wordpress.com/2012/08/08/how-to-install-oracle-database-on-mac-os-any-version/
• https://tech.lds.org/wiki/Oracle_VM

I pulled together some notes as I knew I would be going through the procedure again. Going through that procedure the second time, I encountered the following common error:

TNS:could not resolve the connect identifier specified

I checked to make sure I had set up port forwarding correctly on VirtualBox

I verified that I could telnet to the VM on port 1521 (from my Mac)

telnet 127.0.0.1 1521

I verified that I could successfully log in through sqlplus from the VM

sqlplus my_admin@orcl

I verified that the listener was up (from the VM)

lsnrctl status

I verified that my Mac had a tnsnames.ora file and that it was located at $ORACLE_HOME/network/admin/tnsnames.ora and that read permissions were open.

ORCL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl)
    )
  )

I verified my environment was configured to properly use the sqlplus client on my Mac.

export NLS_LANG="AMERICAN_AMERICA.UTF8"
export ORACLE_HOME=/opt/oracle/instantclient
export DYLD_LIBRARY_PATH=$ORACLE_HOME

Everything seemed to be correct, but I was still getting errors. As I compared the tnsnames.ora on my one Mac that was working correctly and the other, I noticed that the one that wasn't had the entire configuration block indented. I had indented the code in my notes and so when it was copied over, the indent had remained. It couldn't really be that simple, could it?!

It could.

As soon as I removed the indentation so that the service declaration was bumped up directly with the left edge without any spaces, I could successfully connect.

Wow.