If we were any more heroic we’d need capes.
We’ve talked about how fantastic Zapier is in the past (*Spoilers* You can use Zapier to integrate CheddarGetter with a phenomenal number of services), so as you can imagine it makes us pretty pumped to announce that those crazy kids are at it again.
In honor of “the month of love”, they’re adding 28 new apps to Zapier over the next 28 days of February. That means even more CheddarGetter combination possibilities.
You can read the full announcement on their blog here - including a peak at the first batch of three new Zapier-ready apps - and tune in daily for new integration announcements.
Here’s a rundown on what we accomplished in 2013:
- Native Zapier Support - Zapier could be the single most awe-inspiring addition to CheddarGetter in 2013. You can pretty much do anything. Check it out.
- Significant improvements to customer lookup (speed, ease of use, etc).
- Improved processor communication to increase fault tolerance and better handling for general internet glitches
- Various improvements to PayPal Preapproval workflow (more to come!)
- Massive overhaul of database schema including primary key switcheroo on several key tables with zero downtime!
- Billing address info added to email template variables
- Enable customers to enter coupon codes in hosted pages
- Limit promotion application to certain charge types
- Additional billing solution support
- Increased PayPal preapproval limits to $10000 and 3 years
- Resend invoice email receipts via API and GUI
- Improvements to duplicate POST check
- Refactored hook system. It’s not even more powerful and reliable. Added Bill Reminder as a hook event.
- Improvements to credit card validation when payment method update does not result in a revenue transaction
- Additional config options for Bill Reminder events.
- Found and fixed some memory leaks in our background processing
- Several additions to CG’s real-time automated auditor
Now a sneak peek into what we’ve been working on behind the scenes:
- Massive overhaul hosting infrastructure and migration to new hosting provider. Flipping the switch in early 2014!
- Full rebrand and rewrite of marketing site. Easier to use documentation. Coming in Q1 2014.
- New PCI DSS Service Provider Level 1 assessment. Certification expected to be complete in Q1 2014.
- Admin GUI rewrite including significant improvements to dashboard reporting and customer search
- JSON API interface. Read-only metrics in JSON format.
CheddarGetter has long been a proponent of bootstrapped startups. In every single planning meeting, we talk about our core customer base as being the startup. This has caused some degree of trouble for us when talking to larger customers, as they don’t understand how a company like ours can grow and sustain itself by catering to what they see as a fickle, unprofessional group. In our experience, professionalism, talent, and potential have little or nothing to do with a company’s age. It has everything to do with a team’s integrity and standards.
In our search for balance between catering to startups with a low-cost offering, and servicing larger businesses with a powerful and flexible offering, we have made some pricing decisions that have negatively affected our ability to service either.
We currently charge our merchants for any transaction that results in a movement of money from the end customer to the merchant (you, perhaps). That works great if all end customers pay the merchant monthly. In that case, we collect $0.20/customer/month and that covers all of the expenses we incur to service that account over that month plus a little. Unfortunately in many cases, that’s not what happens.
Our current model does not accommodate for other hard costs or high levels of non-revenue generating activity. In other words, other types of services including web hooks and API calls cost money and our current model doesn’t cover for those costs when usage of those services exceeds the norm.
As such, we’re writing this post to start a discussion about pricing for a few valuable services.
In most cases, API calls to CheddarGetter occur fairly infrequently. Businesses who correctly integrate with our API, and use it as it was meant to be used, present a reasonable load on our server, and cause no problems. The cost of providing an API is just a cost of doing business, and we have no desire to charge any more than what is necessary to cover those costs. Most will never pay us directly for a single API call but in the case where a merchant’s usage of the API is significantly disproportionate to the amount of revenue we receive from that customer, we need to do something about it.
With API calls, there may be an additional charge for exceeding a set velocity or exceeding a set total number of API calls in a month. In each case, the limits will be set relative to, say, the number of revenue transactions in a month. We’re not yet doing this but plan to soon. First, we will begin tracking API call quantities and velocities to get a baseline measurement. Based on that data, we will determine what’s normal, then decide at what level to start charging. Our goal here is only to charge for usage well above the average so the vast majority of our customers will not be charged for API calls. Only those with an abnormally high usage will be charged. The fees, if any, will be nominal.
As with API calls, CheddarGetter handles your web hooks with ease in most cases. However when your services hits a snag (something hangs), our service retries that hook multiple times, for multiple seconds each time. This, of course, is a good thing because it makes the hook system tolerant of failure of the listener to process the hook. This can cause significantly elevated usage of resources by our background processes, which can delay other customers’ webhooks if there’s a backlog. This is bad juju, and we feel that good users should not bear the weight of those who are causing the problems.
Similar to API calls, we’re looking at setting some velocity and total limits as well as total time waited for listeners to respond. We’re not doing this yet but plan to soon. Again, only those with abnormally high usage of hooks will be affected.
As you know, transaction charges are a part of life. Some processors charge percentage fees, some charge flat rates, others charge both. We’ve always felt that a flat fee is the most fair method to cover these costs, because it puts a standard value on the process of transacting, rather than sliding number up or down to capture a larger piece of your higher-dollar transactions. In real terms, paying 3% of $100 is a lot harder to swallow than paying $0.20, compared to the difference between paying $0.20 on a $10 transaction vs $0.03. This varies of course depending on your pricing model, but in the end we determined that it was the most equitable model for our customer base.
Transactions are a hard cost for us as well. In the past, we have charged only for Approved (Successful) and Refunded transactions. These are $0.20 for the Blowing Up plan, and $0.25 for the Paypal Only plan. What you haven’t seen are the costs for other transaction types, like Declined, Failed, Authorization (Auth), and Voided. Approved and Refunded transactions are pretty straightforward but the others are not. I’ll explain.
Declined transactions may seem innocuous but these amount to a significant cost to for us. They occur more often than you might think. First, consider repeated failed signups, some are even attempts to fraudulently use one or more stolen cards. Second, consider dunning retries. CG by default continues to attempt to charge the customer 4 more times over 8 days before giving up. You can configure your dunning retry schedule here: https://cheddargetter.com/admin/configuration
Failed is a special transaction that occurs when a transaction is first approved, then later found to be declined. This is typical of ACH (aka echeck) transactions. This mostly occurs when a PayPal payment is backed by a bank draft instead of a credit card. The transaction is initially approved but PayPal then later informs us that the money could not be taken from the PayPal account holder’s bank account.
Authorizations and Voids apply only to those merchants using a validation transaction on signup or payment method change events. When a validation transaction is configured, CG issues a small authorization transaction which tests the card all the way to the issuer including CVV verification (if applicable) and AVS checks (if applicable). This is extremely valuable for those who accept payment methods with a delayed initial bill and for payment method changes mid billing cycle. Without them, you have to trust the customer to enter all of their information properly without validation other than basic format checks. If the payment information is wrong at bill time, it will be declined. That’s a hassle for everyone.
Voids are related to the Authorization. When an Authorization is successful, we need to issue a Void to indicate that we never intend to capture that small Authorization. If this isn’t done, the Authorization will remain on the cardholder’s card statement for several weeks. Hanging Authorizations are a no-no for that reason but also because the banks consider them to be a liability when there’s a large quantity of hanging auths.
In order to continue providing a great service, we felt that it was necessary and fair to pass these costs to the people who are using them.
For most of our customers, these costs will amount to a very small increase on their monthly bill.
For CheddarGetter, the sum of these costs across our entire customer base accounts for a very significant expense each month. That is money that we could and should be using to shore up our service, provide higher availability, and generally make our system stronger and better.
We know that this is a touchy subject.
We know that people don’t like to pay for something that they didn’t realize they were getting for free.
We also know that our competitors already charge for these things, and it is possible that some of our customers made the decision to use CheddarGetter because we did not.
To alleviate potential future discomfort for all involved, here is our plan.
As of yesterday, we turned on the charge system for these “other” transactions (Auths, Voids, Declines, and Failures) on the PayPal Only and Blowing Up plans (legacy plans are not affected). Everyone is able to see these pending transaction charges, if any, on the Billing page: https://cheddargetter.com/admin/billing.
As promised, we are giving all of you the option to tell us you don’t want to pay for them. If you are on a super-tight budget and simply cannot afford the extra charges, we understand. Or maybe you don’t think that these charges should be passed on to customers on principle. Either way, let us know and we will adjust your account so you will not be charged for those transactions.
Now: No change
Soon: Plan to charge for extreme usage / Costs TBD but will be minimal
Take action: This topic is open for discussion here (http://support.cheddargetter.com/discussions/questions/5850-api-and-webhooks-overage-pricing).
Now: No change
Soon: Plan to charge for extreme usage / Costs TBD but will be minimal
Take action: This topic is open for discussion here (http://support.cheddargetter.com/discussions/questions/5850-api-and-webhooks-overage-pricing).
Now: Charge for all transaction types on current plans (legacy plans, Basic, Advanced, Premium, etc, are not affected)
Soon: No further changes
Take action: Review your charges as they accrue here: https://cheddargetter.com/admin/billing. Evaluate how they affect your budget in relation to the benefits to the health of CheddarGetter’s system. If acceptable, no action needed. If you choose to opt out, let us know here (email@example.com).
We’ve had two episodes of lengthy downtime in the past 60 days. Early this morning we were down for almost 3 hours. Back in February it was even longer at over 4 hours on the night of the 6th. Let me first be very clear: Any downtime at all is unacceptable, planned or otherwise. Our job is to do our best to prevent this sort of thing as much as is possible. We’re currently failing in that effort.
Both of these episodes were avoidable. No, it wasn’t a “perfect storm”. Port 443 was hung and we were asleep. In both instances, we were back up within a few minutes of cracking an eyelid.
No, we don’t have our humans monitoring services 24/7. We should. No, we don’t have enough redundancy to avoid this particular issue. We should. Why don’t we? The answer is simple and not one that anyone wants to hear. A choice has been made to keep our hosting infrastructure in it’s current state, a vulnerable state, until we secure additional capital to improve it. We should have more humans and more redundancy. Both cost money and lots of it.
You’re probably wondering what it is we *are* doing. I’ll give you an example. Yesterday I posted a long overdue change log of what the development team has been doing over the past few months. In one of those many bullet points, I mentioned that we had flipped our database keys - primary keys and foreign keys, from a 128 byte UUID to a basic integer. One of our favorite customers emailed me and said:
Impressive! Any quick tips you can share for low impact DB maintenance?
It’s quite an unsettling feeling that we are now boasting about DB maintenance fu when we just had a ridiculous amount of downtime. Well, this is actually a great example of what we have been doing to improve and how we sacrifice our time in extra effort to avoid any time offline. It’s interesting, too, and a subject of a blog post that has never surfaced. So, I’ll just describe briefly what we did and how we did it.
Before I get started, I should explain that the reason we were forced to make this drastic change to our database schema was another fault of our own. When I first created the schema for CheddarGetter’s primary database, I decided to give UUIDs a whirl. I used them for primary keys and foreign keys. The columns are char(36). They’re hex UUIDs with dashes — the kind you get from MySQL’s UUID() function. That’s 128 bytes UTF-8. Yeah, UTF-8, entirely unecessary for these columns. Integers are far more efficient in BTREE indexes than are 128 byte strings and far smaller, too. The difference in efficiency was causing severe headaches for everyone — our dev and support teams and for our customers. We absolutely had to make this change.
The size of our database at the time was around 12G. That’s not huge but it’s large enough that making the changes we needed to make would cause hours of planned downtime if we were to use traditional means. Since outages are unacceptable, even those for planned maintence, we figured out a way to do it without affecting live services at all, not even for a minute. Instead of a few days of work, it took more than the entire month of January. We put in the extra time in order to prevent hours, likely >10 hours of database locks. Here’s how we did it:
- Plan the new schema
- New id columns
- New primary keys
- New foreign keys
- Override database access in the app framework so the app is tolerant of changes
- Make inserts create both keys
- Make updates reconcile both keys
- Make lookups by primary key work for either key
- Make the changes
Normally when you make schema changes, you simply use an ALTER statement. The trouble with ALTERs is that the table is locked while the alter takes place. When an ALTER is executed, the entire table must be recreated. For a single 4G table, for example, that can take a long time, maybe an hour or more. Ours was a unique situation because we were going to have to run an ALTER more than once for the same table because we were dealing with primary and foreign keys. In short, these changes cascade into other tables and had to be well orchestrated. Each table had to be altered, then each related table had to be altered, then the original table altered again, and the related tables again, too. That’s craziness, so we found a way to do it without any downtime.
Enter Percona Toolkit's pt-online-schema-change, a well-named tool. An ALTER using pt-online-schema-change 1) creates a copy of the target table schema, 2) runs the ALTER on the new empty table, 3) copies all the rows from the old table, 4) ensures the two tables are in sync, 5) replaces the old table with the new. There’s a lot of database trigger witchcraft mixed in there and it works quite well. The sacrifice is time. It takes around 10x longer to alter a table using pt-online-schema-change than a traditional alter.
Here’s a simple example of what we needed to do and how we did it with pt-online-schema-change:
Consider this simple schema:
CREATE TABLE Caves ( id char(36) not null primary key, location varchar(255), createdDatetime datetime ) ENGINE=InnoDB; CREATE TABLE Bears ( id char(36) not null primary key, caveId char(36) not null, constraint caveId_fk foreign key (caveId) references Caves(id), name varchar(255), createdDatetime datetime ) ENGINE=InnoDB;
First, alter the table remove the primary key, add the new primary and set an index on the old primary:
alter table Caves add unique index id(id), drop primary key, add column newId int unsigned not null primary key auto_increment FIRST;
… and the pt-online-schema-change equivalent (two steps required):
pt-online-schema-change --alter \ "add column newId int unsigned not null \ auto_increment FIRST, add index newId(newId)" \ D=testdb,t=Caves --lock-wait-time=50 \ --alter-foreign-keys-method auto \ --nocheck-replication-filters --dry-run pt-online-schema-change --alter \ "drop primary key, drop index newId, \ add primary key(newId), add unique index id(id)" \ D=testdb,t=Caves --lock-wait-time=50 \ --alter-foreign-keys-method auto \ --nocheck-replication-filters --dry-run
Next, do the same for the related table(s) but also make foreign key change(s):
alter table Bears add unique index id(id), drop primary key, add column newId int unsigned not null primary key auto_increment FIRST, add column caveNewId int unsigned not null after id;
pt-online-schema-change --alter \ "add column newId int unsigned not null \ auto_increment FIRST, add index newId(newId)" \ D=testdb,t=Bears --lock-wait-time=50 \ --alter-foreign-keys-method auto \ --nocheck-replication-filters --dry-run pt-online-schema-change --alter \ "drop primary key, drop index newId, \ add primary key(newId), add unique index id(id), \ add column caveNewId int unsigned after id" \ D=testdb,t=Bears --lock-wait-time=50 \ --alter-foreign-keys-method auto \ --nocheck-replication-filters --dry-run
Populate the new foreign key:
update Bears b, Caves c set b.caveNewId=c.newId where b.caveId=c.id;
We did this a block at a time using the `watch` command:
watch -n 0.1 'mysql -e "select count(*) from Bears \ where caveNewId is null; update Bears b \ set b.newCaveId=(select b.newId from Bears b \ where b.caveId=c.id) \ where b.newCaveId is null limit 50000;" testdb'
Last, setup the new foreign key(s):
alter table Bears add constraint cave_fk foreign key (caveNewid) \ references Caves(newId);
pt-online-schema-change --alter \ "cave_fk foreign key (caveNewid) \ references Caves(newId)" D=testdb,t=Bears \ --lock-wait-time=50 --alter-foreign-keys-method auto \ --nocheck-replication-filters --dry-run
For one of our tables, that process took 6 days.
Alright, long story short, we’re working hard to improve and to avoid as much downtime as possible. Things are going to get better. Honest.
Zapier is a sort of universal adapter for APIs. We have worked with their team to integrate CheddarGetter’s API in the most comprehensive way possible. This means that you can use Zapier to connect your CheddarGetter account to a ton of other services across the web.
If you want to try out some “zaps”, sign up for an account using our coupon code and you can get a 50% discount if you decide to upgrade to their paid plan. It’s free to start.
Zapier Integration Recipes
Here are a few example recipes using some of the services we regularly use at CheddarGetter:
- New Subscription to SMS
- New Subscription to MailChimp
- New Subscription Email Draft
- New Subscription to Basecamp Message
- New Subscription to Highrise Contact
- New Transaction to Ambassador
If you don’t see what you need, you can make your own “Zaps” very easily with their simple editor.
Try some of these combinations:
- CheddarGetter with Freshbooks
- CheddarGetter with Quickbooks
- CheddarGetter with Zendesk
- CheddarGetter with Magento
- CheddarGetter with Salesforce
Zapier Is Open
The best thing about integrating with Zapier? It’s completely open. This means that any service you happen to use is a potential integration for CheddarGetter. You can scan their giant list of companies that are already integrated with Zapier, and if you find something you like, create a recipe between CheddarGetter and the system of your choosing.
Open also means that if your service is not on that list, you can add it yourself. That requires a bit of coding on your part, but if it’s something you think is worthwhile, chances are someone else will too. Once you write an integration, it’s available for all other Zapier users. Good karma all around.
As of Sunday 2013-01-13, we’re living on a new, higher-powered master database server.
We’ve been planning to make this move for some time. Our database had grown too large to hold in RAM on our previous server, and some queries were taking upwards of a minute to complete. That was unacceptable, so we began the process of building out a bigger server, with architecture improvements to allow us to scale more easily. The new server had been setup for about 6 days and was happily replicating our production data in real time. We had run almost all of our pre-migration tests, and we were planning to make the switch this week.
In the wee hours of Sunday morning EST, our hand was forced. Our old database server failed, and we were compelled to make the switch to the new server early. There were a couple glitches during the switchover that caused the database to be down longer than it should have been. For that we’re very sorry. I mentioned that we had run *almost* all of our tests. It turns out that the ones we hadn’t gotten to would have shown us a potential problem earlier.
In the end though, the cutover went smoothly, and we’re now up and running full steam on a vastly improved master server.
On the agenda this year is a project to improve our database layer. Today we have a single point of failure at the database. That’s obviously not ideal. Worst case scenario, we could again experience unplanned downtime of a couple of hours or more. But the potential downtime grows as the size of the database grows. We maintain at all times verified real-time replicated data on a passive slave to be ready for such a catastrophe. Our current architecture requires that human interaction take place to fail over to the slave server. We run tests to determine if there is data loss and point the app to the new database host. That process can take some time.
With the database layer improvement project, we will gain the benefit of increased hardware resources and an architecture that ideally eliminates the possibility of downtime caused by failure of a single database node. In other words, the database will no longer be a single point of failure. We’ll have more on this later.
Here is a list of our primary development projects from last year, excluding some minor updates and fixes. If you have questions about anything here, just hit us up in Support!
- Credits, Refunds, and Voids, oh my
- Promotions and Coupons
- Subscription Billable hook
- Invoice Review audit process
- Apple in-app purchase subscription conversion
- Add support for the Beanstream gateway
- API response caching - Big speed and bandwidth improvements!
- Add sales tax information to hosted pages.
- Resend payment, decline, void, refund emails
- More background processing = performance enhancements
- One-time invoice total amount can equal zero
- Improve gateway decline/error response handling
- Improve support system integration
- Remote IP passthru to all gateways
- Search customers by coupon code
- Add marketing metrics in email templates
- Enable add-charge, edit-charge, delete-charge, set-item-quantity GUI and API for outstanding invoice.
- Add waive and run invoice actions for outstanding invoice. Supports autonomous reactivation.
- Opcode caching
- Improve revenue report
- Interactive integration guide.
- PayPal error response improvements
- PayPal WPP undocumented response codes handled
- Fix PayPal standard reactivation after cancellation
- Fix PayPal preapproval limit currency conversion
- Enable PayPal subscription method in admin GUI.
- PayPal third-party permission support — improved preapproval workflow.
- Use memo field in PayPal preapproval to inform customer of subscribed plan information
Here is the full changelog.