Steve Manuel

[This post is my answer to a question asked on Quora. To see the original question, click the link at the bottom of the post.]

The most difficult part of this program is the 15 minute time limitation to process the screenshots.

I was able to render a full (top-to-bottom) screenshot of the webpage response in 4.285 seconds on my Macbook Air (Core i7, 2 GHz, 8GB RAM). It would take my machine 714 minutes to render and save 10,000 screenshots assuming the same time is spent for each one.

The only way (in 2015) to achieve this is to parallelize across as many cores available per machine and distribute the workload across numerous machines. Currently, the largest available machine on Amazon Web Services (AWS) has 36 cores and 244 GiB RAM. I’m not going to try and quantify how much “faster” this is than my machine, but I don’t think it alone is fast enough to process the images in the time you’ve allowed.

Since you tagged this question with ‘Ruby (programming language)’, I’m assuming you’d like to know how it could be done in Ruby.

The gem, 'webshot’, makes screenshot capture very easy. There is a dependency, however, on a tool called 'PhantomJS’, which is basically a web browser without a user interface, primarily used for testing websites and webapps. See the README for webshot for instructions on how to install it and PhantomJS: https://github.com/vitalie/webshot

Once you have the gem and dependencies installed, writing the Ruby is fairly straightforward. Read in your website URLs to capture, and provide them to webshot along with the appropriate dimensions of the screenshot (yes, you can take full-page screenshots). By default, webshot will write the screenshot file to disk after it has been captured. My suggestion would be to upload the file to S3 (or similar) so that all your screenshots are stored in one central location, since you will need to run this program on many separate machines who all have their own disks.

Now, let’s assume you have a working version of your screenshot program on your own computer. The next step would be to deploy it to cloud servers which can share the workload by chipping away at the list of 10,000 websites at the same time. It may not be the most cost-effective, but the easiest way to do this would be on a Platform-as-a-Service (PaaS) like Heroku, where you can push your code once and make copies of the same machine setup, each running your program. You will need to run a sample size of URLs on this system, adding a worker server until you know the optimal number of servers you need to meet the 15 minute requirement. One note: you will need to include a buildpack for PhantomJS if you use Heroku (stomita/heroku-buildpack-phantomjs).

The missing piece is how to supply the 10,000 website URLs to these machines where they each take part of the workload, and so that there isn’t duplication (wasting time and resources), meaning no machine should grab the same URL as another machine. This can be managed using a queue like RabbitMQ (https://www.rabbitmq.com/). This would sit in front of the screenshot workers, which would each read URLs, one-by-one from the queue until all the website screenshots have been captured. An added benefit is that you can provide RabbitMQ the list of URLs dynamically via some interface (iOS app, website form, etc), rather than a static list that you might upload alongside your code. Most PaaS (including Heroku) have click-to-install add ons like RabbitMQ making it simple to deploy alongside your app.

See more answers from the original thread on Quora: https://www.quora.com/If-you-were-asked-to-write-a-script-that-should-take-screenshots-of-10-000-websites-and-it-should-process-in-15-minutes-how-would-you-do-it

[This post is my answer to a question asked on Quora. To see the original question, click the link at the bottom of the post.]

As with most “X vs Y” programming language or framework questions, it depends.

If you’re building a new product and need to iterate quickly based on relatively straightforward CRUD interaction between a user-facing dashboard and a database, then, no, Go is not particularly better than Python/Django or Ruby/Rails. Those frameworks were designed specifically to make building MVC-based web apps easy by eliminating boilerplate.

Where you will, however, see major benefits by using Go over Python or Ruby is in I/O-bound processing. Need to handle thousands of open TCP connections at once? Tying to synchronize data across multiple servers? Shipping a compiled binary to end-users across multiple architectures? Go is definitely going to be a better choice.

Aside from that, other benefits of Go are in its built-in concurrency primitives (channels, goroutines, etc)  which make writing software which utilizes multi-core processors easier, its package / dependency resolution system (see go get, go install as part of the Go toolchain), its notion of interfaces, and the type system. All these features make Go shine, and eliminate a lot of the cruft you might find in other languages.

There is plenty more about Go that sets it apart from languages, but coming from the likes of Python or Ruby, these features may not be what you’re looking for. Go has no “classes” per sé, though structs + interfaces + method receivers get you most of the way there. There is no inheritance in Go, which both Rails and Django take advantage of heavily. Because there is no built in form of generics, you may feel like you are rewriting a lot of code at certain points or abusing reflection at runtime. Lastly, error handling may feel like a major burden to begin with, but most end up seeing the value in it fairly quickly.

If you are looking for a Go web framework that is similar to Rails or Django, Beego is probably your best bet. Others, such as Goji, Gingonic, Martini, et al are going to be a bit more DIY. The current best practice in writing web apps in Go is to use as much of the standard library as possible and plugging the gaps with small utility packages like Gorilla, or more advanced routing packages like httprouter.

See more answers from the original thread on Quora: https://www.quora.com/Is-it-true-that-using-Go-is-a-lot-better-than-using-Python-Django-or-Ruby-Rails

[UPDATE] - the below is now deprecated in favor of a new way to work with Google App Engine. See official documentation here: https://github.com/stripe/stripe-go#google-appengine

You’ve followed Stripes excellent docs step-by-step, but when it comes to making a request to their API, you’re failing with something like this error:

2014/11/24 20:39:44 Requesting POST “/v1/charges”

2014/11/24 20:39:44 Request to Stripe failed: Post https://api.stripe.com/v1/charges: http.DefaultTransport and http.DefaultClient are not available in App Engine. See https://developers.google.com/appengine/docs/go/urlfetch/overview

Conveniently, Stripe developers (at the time of writing this post) have given us a clue. Looking into the “urlfetch” package documentation at the link they provide:

“The convenience function urlfetch.Client returns an *http.Client that uses the urlfetch.Transport”

Since under the hood, the Stripe library needs to make a POST request to the Stripe endpoint, it naturally looks to http.Client for such capability. However, App Engine *does not* support that directly through its net/http package, so that (presumably) the platform can better monitor the bandwidth your app is using.

How to get around this?

Looking into Stripe’s source code for the Go library, there is an exported function called NewInternalBackend() - our savior! This enables us to change the way our app communicates with the Stripe API. This function returns a pointer to an InternalBackend and implements the Backend interface. This allows us to pass our *InternalBackend to another Stripe library function, SetBackend().

Here’s the code:

// we need the current App Engine Context, so make sure you have done this earlier within your handler function
ctx := appengine.NewContext(req) // 'req' is the *http.Request passed to the handler

// passing an empty string will trigger NewInternalBackend() to use the unexported constant "defaultUrl" as the endpoint
gBackend := stripe.NewInternalBackend(urlfetch.Client(ctx), "")
stripe.SetBackend(gBackend)

Including this before making any call to Stripe’s API should clear that error and let App Engine communicate with Stripe.

Page.js is an excellent client-side router modeled after Express router. It’s simple, declarative and easy to read/understand someone else’s work on shared projects.

You’re probably aware of that if you’re reading this - you just want to know how to make it work when a user comes to your-app.com/some-route instead of hitting the root route first! Ok, here’s how I solved that…

In this case, I was using Apache as the web server, so I added the following into the site’s .htaccess file:

# BEGIN ClientRouting

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.html$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.html [L]
</IfModule>

# END ClientRouting

This tells Apache to serve index.html for requests that match /. Then, if we read a request that doesn’t match /, check to see if that file or directory exists. If so, serve it, otherwise, serve index.html

Page.js takes over once the DOM loads, so given that your request path is recognized by your router, it will render whatever you’ve declared in the Page.js callback for that route.

FB Newsroom: link  -  TechCrunch article: link

You can put practically all the Internet’s ad inventory into 4 buckets: Search, Display, Video, and Facebook. Facebook has a lot of ad inventory. A LOT. We’re talking about billions upon billions of impressions per day worth of inventory. That means a lot of ads can be shown, and money be made. However, it also means that Facebook needs to sell TONS of ads.

When advertisers buy Facebook’s inventory, Facebook makes money – only Facebook. Here’s the problem with that – if the amount spent per year on online advertising doesn’t grow at a faster rate, then every additional dollar made by Facebook makes means a dollar lost by a publisher.

To demonstrate this, let’s say a typical ad buyer is going to split their budget up like so: 50% display, 25% video, 5% search, and 20% Facebook. If you’re a publisher and make money from display ads, you don’t want any other media going up, because there’s a good chance it will mean your number goes down. If Facebook continues to show that they have an effective ad product, advertisers aren’t going to spend less money there – especially if advertisers can now target your exact audience on Facebook, for less money! Consider the economic supply and demand curve: more Facebook inventory, lower cost per ad.

Facebook is stealing publishers’ audiences and in return giving them a “Like” button.

Unfortunately, there are only a couple of ways publishers can survive this in the long run. The first (and likely only) way would be to remove Facebook plugins from publishers’ sites. Yes, this means no “Like” button, Facebook Comments, Like Box, Facebook sign-in, or any other copy-and-paste-Facebook-code-onto–your–site-widgets. Those widgets are the main access points to your audience, and for every user you get to your site, you’re adding to Facebook’s targeting power.

The other way would be to convince the vast majority of your site visitors to opt-out of this targeting on Facebook – good luck with that.

Beyond publishers, ad giants like Google, Yahoo, and Aol are the ones who should be concerned most.

Sure, small-time web sites will suffer from losing ad revenue – boo hoo. What about the billion dollar behemoths employing tens of thousands around the world? Not reacting to this as aggressively as possible could mean more than losing their spot on the leaderboard, it could mean collapse.

Doesn’t Google have a lot of inventory and targeting too? They’re scanning emails, observing what videos you watch on YouTube, who’s in your Circles, they know what you’re looking for and where you’re driving! Yes, Google has so much great data and long and far the best ad-targeting available. Google also shares their revenue with publishers and content creators. They are paying the folks who contribute content to the Internet, providing you with articles, jokes, videos, and more. Because of this, Google has to account for that overhead in its cost, which drives the price of ads up just a bit. Since Facebook is the owner of not only the ad network, but the content too (all your pictures, posts, notes, and updates seen by your friends), they don’t need to charge as much to an advertiser. There is no revenue split on Facebook. So publishers, think about it – you’ve posted content to your Facebook page for years and have you ever seen a dime?

Just keep in mind that Facebook, not you, now owns your audience.

Discuss on hacker news & reddit

After building applications with Meteor, I’ve started to observe my design pattern choices changing. More of my Javascript looks like Meteor APIs, and I think that’s great! One example of this is in a package I wrote, GAEvents.

This package, which I recently published, is a small wrapper for Google Analytics Events that is easier to use and looks like Meteor’s Collection API. I find the simplicity and structure of Collections in Meteor to keep my code clean and readable - something I never thought the Events API to be!

When it came time to start tracking some custom events in my Meteor app, I thought it would be a great opportunity to take something ugly and make it better. Here is the wrapper, without the Meteor-specific package code:

var root = this;

root.GAevents = root.GAevents || {}

// just like the collections API, pass an object with your event data
// along with true if the event shouldn't effect bounces (defaults to false)
GAevents.insert = function(obj, bounce) {
    // if site uses ga.js (old)
    if (window._gaq) {
        _gaq.push([
            '_trackEvent',
            obj.category,
            obj.action,
            obj.label || null,
            obj.value || null,
            bounce || false
        ]);
    }
    // if site uses analytics.js (new)
    else {
        ga('send', 'event',
            obj.category,
            obj.action,
            obj.label || null,
            obj.value || null, {
                'nonInteraction': bounce || false
            }
        );
    }

}

// for more information about Google Analytics event tracking:
// old (ga.js): https://developers.google.com/analytics/devguides/collection/gajs/eventTrackerGuide
// new (analytics.js): https://developers.google.com/analytics/devguides/collection/analyticsjs/events

You can download this on Github, or if you’re using Meteorite in a Meteor project, add it with:

    
        $ mrt add GAevents
    

[This post is my answer to a question asked on Quora. To see the original question, click the link at the bottom of the post.]

I think RubyMotion is an excellent tool, don’t get me wrong, but it’s really made for Ruby developers (there are some benefits beyond syntax & structure, editor/IDE agnostic, expressiveness, meta-programming, etc). If your skill levels are practically the same in Obj-c and Ruby, I wouldn’t go the RubyMotion route. As a beginner, while you might be able to get some basic apps up and running faster with RubyMotion, you’ll probably have to dedicate just as much time learning how to implement most of your planned app’s features as you would writing Obj-C in Xcode. While there are some great resources for using RubyMotion, you’ll find a lot more available in Obj-C. In regards to a RubyMotion job market, although this is purely speculative, I’d say there is little chance you’ll find a job building iOS apps in Ruby. In my opinion, businesses who are iOS-driven, or at least have a portion of their interest in mobile, are not going to commit to RubyMotion vs. Obj-C. (You can see a list of applications in RubyMotion’s app directory: RubyMotion - Apps) While there currently is no real downside to building an app in RubyMotion, it is possible that someday Apple will prohibit the use of such toolchains to create apps, although this is how RubyMotion explains it:

Applications submitted to the App Store must conform to the Review Guidelines dictated by Apple. RubyMotion implements a dialect of Ruby that conforms to those rules. RubyMotion apps are fully compiled, do not download or interpret code and are using public iOS APIs through the exact same machinery as regular Objective-C apps.

Additionally, it is possible that the toolchain will not be maintained as it is today (I doubt this will be the case, Laurent & co don’t seem like they’re stopping anytime soon). If this is the case, it may become incompatible with new versions of iOS whose APIs aren’t implemented in RubyMotion. RubyMotion is an amazing tool, and I’d recommend it to anyone already comfortable with the Ruby programming language. It’s great to work with, especially testing, and there are some beautiful apps out there built on it. However, based on the amount of experience you’ve shared with Ruby/Obj-C, I would say it will be more beneficial to you if you can write native applications for iOS in Obj-C.

However, if you are skilled in both Ruby and Obj-C, RubyMotion can be an incredibly powerful tool and a must-try. The ability to create for iOS, yet write & think in Ruby is great for creativity, productivity and fun. If you have experience in both languages, definitely check it out! I didn’t want this post to steer anyone away from learning.

Other RubyMotion items of interest:

• nitron - data like ActiveRecord
• BubbleWrap - helpers/wrappers for more Ruby-like APIs
• Book: RubyMotion: iOS Development with Ruby

See more answers from the original thread on Quora: http://www.quora.com/RubyMotion/Should-new-developers-invest-in-RubyMotion-or-stick-with-Objective-C