First, you need to your certificate and key. For completeness, you want to combine your certificate with the certificate chain provided by your signing authority. In the case of GoDaddy, they gave me two files:
example.com.crtgd_bundle.crtTo create a combined bundle:
1
| |
In addition, you need to make sure that your keyfile does not have a passphrase associated with it, otherwise Nginx will refuse to start up. To remove the passphrase from an existing key file:
1
| |
You need to copy the combined bundle and your key file to your server and place them in a secure location. On a typical CentOS installation, you want to dump the files in /etc/pki/tls/private.
Next, you need to tell Chef to use the custom certificates. Create a file named /etc/chef-server/chef-server.rb, if doesn’t exist already. Add the following lines to it:
1 2 | |
Now, reconfigure Chef server:
1
| |
That will reconfigure Nginx with the new settings (which are stored in /var/opt/chef-server/nginx/etc) and automatically restart Nginx. You can verify that it’s using the correct certificate by running:
1
| |
https://github.com/StudyBlue/chef-asgard
https://github.com/StudyBlue/chef-mater
https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide
There were a few tricky parts that either weren’t documented well or I just found confusing. I have tried to document some of these parts below.
In case you’re as easily confused as I was, Sonatype uses three different terms when talking about pushing jars:
Pushing jars to Maven Central requires that you produce three artifacts:
To automatically generate the second and third artifacts, add the following to the build plugins section of your pom.xml:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | |
Maven Central requires that you sign your released artifacts with GPG. Note that snapshots do not need to be signed. Sonatype has a thorough article on signing your artifacts:
How to Generate PGP Signatures With Maven
The key steps are:
pom.xmlThe first two steps are straightforward. The only trick with the third step is that you only want to sign artifacts during the release process. If you add the maven-gpg-plugin to your main, every single build will get signed and Maven will prompt you for your passphrase with every build. Instead, you can define a Maven profile with a specific name and include the maven-gpg-plugin there.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | |
Following the recommendation from Sonatype, you will remove the <repositories></repositories> section of your pom.xml. Once you do that, Maven no longer knows what server to use when deploying or staging. To compensate, you need to add a distributionManagement section:
1 2 3 4 5 6 7 8 9 10 | |
You need to configure your pom.xml to inherit from the Sonatype Parent POM:
1 2 3 4 5 | |
Assuming you’ve followed the instructions from Sonatype and setup an account, you’re now ready to deploy snapshots. The process is simple:
1
| |
That will automatically clean, build and push your snapshot to Sonatype.
Pushing a release jar to Maven Central consists of two parts: staging and releasing.
In this part we clean everything up; prepare for the build by creating a tag for the release and updating the pom.xml; perform the build; deploy the artifacts to Sonatype.
The process consists of three commands:
1 2 3 | |
The release:prepare step will prompt you for release information. Here is some sample output from the 2.3.0 release of metrics-statsd:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
Once the artifacts are staged on Sonatype’s Nexus, you need to go through a few annoying steps:
That process is well documented (with images) by Sonatype:
Within two hours, Sonatype will push the artifacts to Maven Centrl.
]]>In fact, the more we utilized Graphite, the more data we wanted to push into Graphite. Systems metrics? Check. IO stats? Check. Apache stats? Check. MongoDB stats on a per collection basis? Why not! We quickly found ourselves with over 50,000 metrics and 500GB of data, all of which was getting updated every 15-30 seconds. Unfortunately, we also rapidly hit the limit of how much data we could record on a single instance. Even with a RAID-0 array of 8x EBS volumes on an m2.2xlarge instance, we couldn’t write the data fast enough. The poor instance began spending more than 50% of it’s time in I/O wait and eventually became non-responsive. What we needed was a way to distribute the write load across multiple instances.
Fortunately, Graphite natively supports running in a distributed environment. A successful deployment consists of three key pieces:
Although there are many different ways you can setup your distributed Graphite environment, we chose to take a straight foward approach:
On the carbon-relay server, you need to configure two key files to indicate that this node should run as a relay: carbon.conf and relay.conf.
carbon.conf
[relay]
RELAY_METHOD = consistent-hashing
LINE_RECEIVER_INTERFACE = x.x.x.x
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2004
DESTINATIONS = cache1:2004:a,cache2:2004:a
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = TrueFor simplicity, we’re using the consistent-hashing method of sharding. The advantage to consistent-hasing is that Graphite will automatically split the metrics evenly across all carbon-cache instances. If you add another carbon-cache node, Graphite will start sending data to it. The disadvantage is that you have absolutely no control over which metric goes to which server. If you start with two nodes and then add a third node, Graphite will not automatically rebalance the data.
A couple key things to note:
relay.conf
[default]
default = true
destinations = cache1,cache2This sample relay.conf shows us splitting the data automatically across two servers: cache1 and cache2.
On the carbon-cache servers, we need to configure the carbon-cache daemon to appropriately accept incoming data from the carbon-relay servers.
carbon.conf
[cache]
LOCAL_DATA_DIR = /opt/graphite/storage/whisper/
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 800
MAX_CREATES_PER_MINUTE = 100
LINE_RECEIVER_INTERFACE = y.y.y.y
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = y.y.y.y
PICKLE_RECEIVER_PORT = 2004
CACHE_QUERY_INTERFACE = 127.0.0.1
CACHE_QUERY_PORT = 7002
LOG_UPDATES = FalseThere are several critical settings here. First, the pickle receiver must be configured correctly to allow incoming data from the carbon-relay server. Second, the line receiver must be configured correctly to allow the the webapp to communicate with the other carbon-cache instances.
The other key is configuring the local_settings.py config file for the webapp correctly:
CLUSTER_SERVERS = [ 'cache1','cache2' ]
MEMCACHE_HOSTS = [ 'memcache1.graphite:11211' ]The only mandatory piece of information in local_settings.py is CLUSTER_SERVERS, which must be a Python array containing strings that are the names (or ip address) of every carbon-cache server. You can optionally include an array of memcached instances to improve web performance dramatically. If you choose to utilize memcached, you can either run it locally on each carbon-cache instance or you can run a cluster of one or more memcached servers remotely. In the example above, we’re using a single memcached instance running on a remote server.
IMPORTANT NOTE: The local_settings.py file MUST be readable by the web server user. If you’re running Apache, this is typically the apache user. If the local_settings.py file is NOT readable, you will get very very strange errors.
To get this all working, specific ports must be open on the various firewalls. If you’re using Amazon EC2, your security groups will need to contain rules explicitly allowing incoming traffic on these ports:
As it turns out, you can test your patterns using the grok ruby library. @a1cy demonstrated the basics to me, but I wanted to describe the full process.
On my machine, I chose to install Ruby 1.9.2 via RVM, but you can use whatever technique you want. If you try to use Ruby 1.8.7, you will get a syntax error when you load the grok gem.
You want to use the “grok-pure” library, which is 100% ruby, rather than “grok”, which depends on a C library.
require 'rubygems'
require 'grok-pure'
grok = Grok.newSince the grok filter in Logstash depends heavily on pattern files, I recommend you download the standard patterns from Github. Either the standard set for Logstash:
https://github.com/logstash/logstash/tree/master/patterns
Or you can use the ones with grok:
https://github.com/jordansissel/ruby-grok/blob/master/patterns/pure-ruby
Dump these into a directory so that you can load them with grok.
grok.add_patterns_from_file("/path/to/patterns/grok-patterns")You can use the same technique to load your own pattern files. Or you can manually add patterns by name:
grok.add_pattern("foo", ".*foo.*")
grok.add_pattern("bar", ".*bar.*")So, you have successfully loaded a set of patterns. How do you test something? Let’s take a sample log line from a MongoDB logfile:
Tue May 15 11:21:42 [conn1047685] moveChunk deleted: 7157
Let’s say we want to grab the date/time as well as the number of the deleted chunk. Unfortunately, the date/time format doesn’t match any of the stock patterns. We can build up a new date pattern and test it each step of the way:
text = "Tue May 15 11:21:42 [conn1047685] moveChunk deleted: 7157"
pattern = '%{DAY}'
grok.compile(pattern)
grok.match(text)The last line returns false if the pattern fails to match the input text. If the pattern matches the input text, Grok.match() returns a Grok::Match object that contains a lot of useful information about the match. You can ask Grok to output the text it capture by using the captures() method:
grok.match(text).captures()
=> {"DAY"=>["Tue"]}You can also test named captures, the same way you would use them in the Logstash grok filter:
pattern = '%{DAY:day_of_week}'
grok.compile(pattern)
grok.match(text).captures()
=> {"DAY:day_of_week"=>["Tue"]}The full date/time pattern for this example looks like the following:
pattern = '%{DAY} %{MONTH} %{MONTHDAY} %{TIME}'
grok.compile(pattern)
grok.match(text).capture()
=> {"DAY"=>["Tue"], "MONTH"=>["May"], "MONTHDAY"=>["15"], "TIME"=>["11:21:42"], "HOUR"=>["11"], "MINUTE"=>["21"], "SECOND"=>["42"]}Now let’s say we want to re-use this date/time pattern and still extract the chunk id from the log line. We can add a named pattern to the grok object manually and then reuse it, the same way we used patterns from the files:
grok.add_pattern("MONGO_TIMESTAMP", '%{DAY} %{MONTH} %{MONTHDAY} %{TIME}')
grok.compile("%{MONGO_TIMESTAMP}")
pattern = '%{MONGO_TIMESTAMP:timestamp}'
grok.compile(pattern)
grok.match(text).captures()
=> {"MONGO_TIMESTAMP:timestamp"=>["Tue May 15 11:21:42"], "DAY"=>["Tue"], "MONTH"=>["May"], "MONTHDAY"=>["15"], "TIME"=>["11:21:42"], "HOUR"=>["11"], "MINUTE"=>["21"], "SECOND"=>["42"]}We still want the chunk id, so we can use additional patterns:
pattern = '%{MONGO_TIMESTAMP:timestamp}%{GREEDYDATA} moveChunk deleted: %{NUMBER:chunk_id}'
grok.compile(pattern)
grok.match(text).captures()
=> {"MONGO_TIMESTAMP:timestamp"=>["Tue May 15 11:21:42"], "DAY"=>["Tue"], "MONTH"=>["May"], "MONTHDAY"=>["15"], "TIME"=>["11:21:42"], "HOUR"=>["11"], "MINUTE"=>["21"], "SECOND"=>["42"], "GREEDYDATA"=>[" [conn1047685]"], "NUMBER:chunk_id"=>["7157"], "BASE10NUM"=>["7157"]}Let’s say we wanted to grab that connection number inside the square brackets:
pattern = '%{MONGO_TIMESTAMP:timestamp} \\[conn%{NUMBER:connection}\\] moveChunk deleted: %{NUMBER:chunk_id}'
grok.compile(pattern)
grok.match(text).captures()
=> {"MONGO_TIMESTAMP:timestamp"=>["Tue May 15 11:21:42"], "DAY"=>["Tue"], "MONTH"=>["May"], "MONTHDAY"=>["15"], "TIME"=>["11:21:42"], "HOUR"=>["11"], "MINUTE"=>["21"], "SECOND"=>["42"], "NUMBER:connection"=>["1047685"], "BASE10NUM"=>["1047685", "7157"], "NUMBER:chunk_id"=>["7157"]}require 'rubygems'
require 'grok-pure'
grok = Grok.new
text = "Tue May 15 11:21:42 [conn1047685] moveChunk deleted: 7157"
pattern = '%{MONGO_TIMESTAMP:timestamp} \\[conn%{NUMBER:connection}\\] moveChunk deleted: %{NUMBER:chunk_id}'
grok.compile(pattern)
grok.match(text).captures()
=> {"MONGO_TIMESTAMP:timestamp"=>["Tue May 15 11:21:42"], "DAY"=>["Tue"], "MONTH"=>["May"], "MONTHDAY"=>["15"], "TIME"=>["11:21:42"], "HOUR"=>["11"], "MINUTE"=>["21"], "SECOND"=>["42"], "NUMBER:connection"=>["1047685"], "BASE10NUM"=>["1047685", "7157"], "NUMBER:chunk_id"=>["7157"]}You can also find an ultra simple example in the grok example folder on Github: https://github.com/jordansissel/ruby-grok/blob/master/examples/test.rb.
]]>Specific disadvantages to MySQL:
MySQL offers the following advantages over PostgreSQL:
First, I added a new mime-type for BlackBerry by adding the following line to config/initializers/mime_types.rb:
1
| |
Next, I added two utility methods to app/controllers/application.rb:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | |
With that in hand, it’s easy to render BlackBerry specific content on specific pages:
1 2 3 4 5 6 | |
kill -9 to stop it. The standard init scripts I had seen didn’t solve this problem at all.
Laliluna has a great article that focuses on RedHat, CentOS and Fedora. Unfortunately, their scripts didn’t work correctly under Ubuntu 8.04 LTS. As a result, I spent some time modifying their scripts to get them to work correctly under Unbuntu. Many thanks to Laliluna for doing the heavy work.
Without any further ado, here’s what I put together:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | |
Here’s a simple approach using a mixin module:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | |
The Enum mixin depends on a NameValuePair class to hold the data:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
I included first and last methods to better support the select and options_for_select helper methods in Rails. Here’s how you might use it:
1 2 3 4 5 6 7 8 9 10 11 12 | |
The discussion centered around article by Joel Spolsky (of Joel on Software) titled Things You Should Never Do, Part 1. His points against rewrites include:
Joel further argues that there are three major reasons developers want to rewrite code and none of them require rewrites:
These are all excellent points. On some level, I agree with this entirely. Even many nasty combinations of all three problems can be solved by steady refactorings. I have worked for places where people pushed for rewrites that weren’t necessary. But these were larger businesses with a well established core product. These were not early startups. That’s why I believe Joel makes several assumptions which are fatal to his arguments.
First, he assumes the software project is really large and complex. While some of us may have worked on projects of that size and scope, quite of few of us work on much smaller projects. Simply put, it’s a matter of scale.Second, as a corollary of his first assumption, Joel also assumes that a rewrite requires years not months. Again, this is likely true for a product like Excel or Word… but this simply isn’t true for many of the sites and products I’ve worked on. Furthermore, the use of agile or rapid development technologies such as Ruby on Rails can dramatically shrink this window.
Third, and perhaps most importantly, Joel assumes that the time required to cope with a messy code-base and make steady refactorings is significantly less than the time required to rewrite the app. And he assumes that’s a worthwhile trade off. This may be clear cut for larger products or companies, but I question whether or not that’s accurate for a startup. The more tangled your code, the longer it takes you to make changes. The longer it takes to make changes, the less nimble you are and the longer it takes you to respond to changes in company direction or marketplace demands.
It’s that last point that I believe is most important to those of us working for small startups. We tend to be small young companies who are still striving to find our exact place in the wider world. We’re often in cutting edge spaces where there is no clear cut path to success. And usually we’re steadily seeing greater numbers of competitors in our space. It seems to me that agility is vitally important to people us. We need to be able to makes changes rapidly as our knowledge of the space evolves. Fundamentally, I think it’s better to have a decent product/feature/whatever out in the hands of consumers than it is to have a nearly perfect product that’s still under development. Don’t get me wrong, I’m sure I’m preaching to the choir. :) But, I think it’s critical to keep the need for agility and nimbleness in the forefront of our thoughts.
Fourth, Joel assumes any architectural problems can be solved by steady refactoring. Frankly, I disagree. I think there exist serious architectural flaws, especially related to scalability that cannot be easily solved by refactoring. eBay, LinkedIn, Facebook and Yahoo have all had major rewrites in their history that were directly attributed to serious architectural failings.
That is not to say that a full rewrite is necessarily a desirable goal. :) However, it takes careful management and planning to avoid finding yourself in this position. eBay used to employ a strategy they called headroom, which basically set aside 20+% of all development time to refactor code and it keep it in top working order. While I think it may very difficult to employ such a strategy in a startup, it may be worth considering.
]]>This time, the folks at Redmond broke how Internet Explorer passes the flashvars parameter into Flash. Typically, when placing a Flash object on an HTML page, you use the following syntax:
1 2 3 4 5 6 7 8 9 10 | |
The OBJECT tag is used by Internet Explorer and the EMBED tag is used by everyone else (Firefox, Safari, etc.). Although the flashvars paramter is not formally described in the HTML 4.0.1 spec, this code worked fine with IE 6 and IE 7. Unfortunately, IE 8 does not pass flashvars into the Flash Player. The only work around is to pass the flashvars parameter as part of the movie name parameter, as shown below:
1 2 3 4 5 6 7 8 9 10 | |
Initially I thought it was trivial: create an IFrameElement and call setInnerHTML.
1 2 | |
Unfortunately, that doesn’t work. It doesn’t cause any errors, but it doesn’t actually fill in the iframe. Instead, you have to use native javascript to write into the document object for the iframe:
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
Voila! However, one of the side effects is that the iframe doesn’t include any CSS unless your embedded HTML references a stylesheet. If you want to manually add a reference to a specific stylesheet, you can do that through native javascript as well:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | |
There’s still one more problem. You can’t use either of native methods until the IFRAME element has been attached to the DOM. The easiest way around this is to add the IFRAME element to a panel and over the onLoad() method for the panel:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
I got the idea and the important code from an article titled Inject HTML into an IFrame from Software As She’s Developed.
]]>1 2 3 4 5 | |
Although I was positive the file(s) were up to date, I ran an svn up just to be safe:
1 2 | |
Nothing changed. If I tried to commit again, I would get the same out of date error. Eventually I figured out that if I delete the folder containing the problematic file, ran svn up to restore the working copy and re-edited (or restored from a backup) the modified file, the problem went away. This suggested an issue with the Subversion metadata, but this blog post from Pageworthy describes a much easier work around:
Delete the all-wcprops file from the .svn metadata folder for the offending file.
In other words, if Subversion complains about src/com/mycompany/Foo.java, then delete src/com/mycompany/.svn/all-wcprops.
xorg.conf just to get dual monitor support to work under Linux. Somehow I never knew about xrandr, KRandR, etc.
ThinkWiki has a great article about using XRandR that really helped me out. When I first launched Kubuntu the two screens were mirrored. So, I tried to use KRandR to set the screens side by side. Unfortunately, it seemed to be confused - it saw there were two outputs, but it thought they were the same screen. This is where the ThinkWiki article came in handy.
With the later versions of Xorg, Ubuntu doesn’t even include an xorg.conf. Instead of even monkeying around with Xorg, I just used xrandr. There were three key steps:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
This shows I have two main outputs connected right now: VGA1 and LVDS1. Under Ubuntu 9.04 (Jaunty), my outputs were labeled VGA and LVDS.
Once we know the name of the outputs, we can disable the secondary output. This is the key step to getting a large virtual desktop working without something like Xinerama. If you don’t disable the secondary output, Xorg never seems to be able to successfully distinguish between the two outputs.
1
| |
When we re-enable the outputs we can specify the location of the secondary display, relative to the primary. We can also let xrandr figure out the best resolution for each:
1
| |
Voila. Finally, if you want to automate this, the ThinkWiki article has a great little script you can use. However, I did have to modify it slightly … I had to force the VGA1 output off before setting them both to auto. Without that, the secondary screen remained blank.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | |
Place this in /etc/X11/Xsession.d as 45custom_xrandr-settings and it will automatically run.
PropertyPlaceHolderConfigurer beans declared in two separate XML files. Unfortunately, the second property file wasn’t getting loaded, so we would get errors about Spring being unable to resolve particular properties.
I found this great post summarizing the solution:
http://dotal.wordpress.com/2007/09/14/mulitple-propertyplaceholderconfigurer-configurations/
Basically, you have to update the definition for the first PropertyPlaceHolderConfigurer bean that gets loaded and add the ignoreUnresolvablePlaceholders property:
1 2 3 4 | |
1
| |
Of course, I also loved the final suggestion in Case Three: using a has_many with a condition on it:
1
| |
Excellent!
]]>ActionRedirect class instead.
Here’s a basic example from inside the execute method in an Action.
1 2 3 4 5 | |
The ActionRedirect class is basically the same as my ForwardParameter class and is probably preferable.
My concern is that in a large project, the use of a “context” may make it difficult to figure out which specific message code is used in a given place. The generic exception handling is especially beneficial for the “this should never happen” cases where there isn’t much the app can do about the exception, but should still handle it in a robust manner.
]]>struts-config.xml:
1
| |
Unfortunately, this doesn’t provide a mechanism for passing request parameters to the target. So what can you do if you want to redirect to another page and pass some parameters along? You use an additional class:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | |
Here are some examples of using ForwardParameters:
Long form:
1 2 3 4 5 | |
Concise form:
1 2 | |
Ultra concise form:
1
| |
This is based on a suggestion I found on Experts-Exchange by dualsoul.
Edit June 5th, 2006:
As someone just pointed out to me this morning, you can use the ActionRedirect class instead of my ForwardParameters class. According to the JavaDocs, ActionRedirect was added in Struts 1.2.7 and is:
]]>… a subclass of ActionForward which is designed for use in redirecting requests, with support for adding parameters at runtime.
The article is available on oreillynet.com:
What Is Prefactoring? by Ken Pugh, author of Prefactoring.
]]>