Balancing Usage Levels and Charges

CheddarGetter has long been a proponent of bootstrapped startups. In every single planning meeting, we talk about our core customer base as being the startup. This has caused some degree of trouble for us when talking to larger customers, as they don’t understand how a company like ours can grow and sustain itself by catering to what they see as a fickle, unprofessional group. In our experience, professionalism, talent, and potential have little or nothing to do with a company’s age. It has everything to do with a team’s integrity and standards.

In our search for balance between catering to startups with a low-cost offering, and servicing larger businesses with a powerful and flexible offering, we have made some pricing decisions that have negatively affected our ability to service either.

We currently charge our merchants for any transaction that results in a movement of money from the end customer to the merchant (you, perhaps). That works great if all end customers pay the merchant monthly. In that case, we collect $0.20/customer/month and that covers all of the expenses we incur to service that account over that month plus a little. Unfortunately in many cases, that’s not what happens.

Our current model does not accommodate for other hard costs or high levels of non-revenue generating activity. In other words, other types of services including web hooks and API calls cost money and our current model doesn’t cover for those costs when usage of those services exceeds the norm.

As such, we’re writing this post to start a discussion about pricing for a few valuable services.

API Calls

In most cases, API calls to CheddarGetter occur fairly infrequently. Businesses who correctly integrate with our API, and use it as it was meant to be used, present a reasonable load on our server, and cause no problems. The cost of providing an API is just a cost of doing business, and we have no desire to charge any more than what is necessary to cover those costs. Most will never pay us directly for a single API call but in the case where a merchant’s usage of the API is significantly disproportionate to the amount of revenue we receive from that customer, we need to do something about it.

With API calls, there may be an additional charge for exceeding a set velocity or exceeding a set total number of API calls in a month. In each case, the limits will be set relative to, say, the number of revenue transactions in a month. We’re not yet doing this but plan to soon. First, we will begin tracking API call quantities and velocities to get a baseline measurement. Based on that data, we will determine what’s normal, then decide at what level to start charging. Our goal here is only to charge for usage well above the average so the vast majority of our customers will not be charged for API calls. Only those with an abnormally high usage will be charged. The fees, if any, will be nominal.

Web Hooks

As with API calls, CheddarGetter handles your web hooks with ease in most cases. However when your services hits a snag (something hangs), our service retries that hook multiple times, for multiple seconds each time. This, of course, is a good thing because it makes the hook system tolerant of failure of the listener to process the hook. This can cause significantly elevated usage of resources by our background processes, which can delay other customers’ webhooks if there’s a backlog. This is bad juju, and we feel that good users should not bear the weight of those who are causing the problems.

Similar to API calls, we’re looking at setting some velocity and total limits as well as total time waited for listeners to respond. We’re not doing this yet but plan to soon. Again, only those with abnormally high usage of hooks will be affected.

Transactions

As you know, transaction charges are a part of life. Some processors charge percentage fees, some charge flat rates, others charge both. We’ve always felt that a flat fee is the most fair method to cover these costs, because it puts a standard value on the process of transacting, rather than sliding number up or down to capture a larger piece of your higher-dollar transactions. In real terms, paying 3% of $100 is a lot harder to swallow than paying $0.20, compared to the difference between paying $0.20 on a $10 transaction vs $0.03. This varies of course depending on your pricing model, but in the end we determined that it was the most equitable model for our customer base.

Transactions are a hard cost for us as well. In the past, we have charged only for Approved (Successful) and Refunded transactions. These are $0.20 for the Blowing Up plan, and $0.25 for the Paypal Only plan. What you haven’t seen are the costs for other transaction types, like Declined, Failed, Authorization (Auth), and Voided. Approved and Refunded transactions are pretty straightforward but the others are not. I’ll explain.

Declined transactions may seem innocuous but these amount to a significant cost to for us. They occur more often than you might think. First, consider repeated failed signups, some are even attempts to fraudulently use one or more stolen cards. Second, consider dunning retries. CG by default continues to attempt to charge the customer 4 more times over 8 days before giving up. You can configure your dunning retry schedule here: https://cheddargetter.com/admin/configuration

Failed is a special transaction that occurs when a transaction is first approved, then later found to be declined. This is typical of ACH (aka echeck) transactions. This mostly occurs when a PayPal payment is backed by a bank draft instead of a credit card. The transaction is initially approved but PayPal then later informs us that the money could not be taken from the PayPal account holder’s bank account.

Authorizations and Voids apply only to those merchants using a validation transaction on signup or payment method change events. When a validation transaction is configured, CG issues a small authorization transaction which tests the card all the way to the issuer including CVV verification (if applicable) and AVS checks (if applicable). This is extremely valuable for those who accept payment methods with a delayed initial bill and for payment method changes mid billing cycle. Without them, you have to trust the customer to enter all of their information properly without validation other than basic format checks. If the payment information is wrong at bill time, it will be declined. That’s a hassle for everyone.

Voids are related to the Authorization. When an Authorization is successful, we need to issue a Void to indicate that we never intend to capture that small Authorization. If this isn’t done, the Authorization will remain on the cardholder’s card statement for several weeks. Hanging Authorizations are a no-no for that reason but also because the banks consider them to be a liability when there’s a large quantity of hanging auths.

In order to continue providing a great service, we felt that it was necessary and fair to pass these costs to the people who are using them.

For most of our customers, these costs will amount to a very small increase on their monthly bill.

For CheddarGetter, the sum of these costs across our entire customer base accounts for a very significant expense each month. That is money that we could and should be using to shore up our service, provide higher availability, and generally make our system stronger and better.

We know that this is a touchy subject.

We know that people don’t like to pay for something that they didn’t realize they were getting for free.

We also know that our competitors already charge for these things, and it is possible that some of our customers made the decision to use CheddarGetter because we did not.

So…

To alleviate potential future discomfort for all involved, here is our plan.

As of yesterday, we turned on the charge system for these “other” transactions (Auths, Voids, Declines, and Failures) on the PayPal Only and Blowing Up plans (legacy plans are not affected). Everyone is able to see these pending transaction charges, if any, on the Billing page: https://cheddargetter.com/admin/billing.

As promised, we are giving all of you the option to tell us you don’t want to pay for them. If you are on a super-tight budget and simply cannot afford the extra charges, we understand. Or maybe you don’t think that these charges should be passed on to customers on principle. Either way, let us know and we will adjust your account so you will not be charged for those transactions.

In Summary

API Calls Now: No change
Soon: Plan to charge for extreme usage / Costs TBD but will be minimal
Take action: This topic is open for discussion here (http://support.cheddargetter.com/discussions/questions/5850-api-and-webhooks-overage-pricing).

Web Hooks Now: No change
Soon: Plan to charge for extreme usage / Costs TBD but will be minimal
Take action: This topic is open for discussion here (http://support.cheddargetter.com/discussions/questions/5850-api-and-webhooks-overage-pricing).

Transactions Now: Charge for all transaction types on current plans (legacy plans, Basic, Advanced, Premium, etc, are not affected)
Soon: No further changes
Take action: Review your charges as they accrue here: https://cheddargetter.com/admin/billing. Evaluate how they affect your budget in relation to the benefits to the health of CheddarGetter’s system. If acceptable, no action needed. If you choose to opt out, let us know here (team@cheddargetter.com).

Database Craziness. Downtime, too. Not Directly Related.

We’ve had two episodes of lengthy downtime in the past 60 days. Early this morning we were down for almost 3 hours. Back in February it was even longer at over 4 hours on the night of the 6th. Let me first be very clear: Any downtime at all is unacceptable, planned or otherwise. Our job is to do our best to prevent this sort of thing as much as is possible. We’re currently failing in that effort.

Both of these episodes were avoidable. No, it wasn’t a “perfect storm”. Port 443 was hung and we were asleep. In both instances, we were back up within a few minutes of cracking an eyelid.

No, we don’t have our humans monitoring services 24/7. We should. No, we don’t have enough redundancy to avoid this particular issue. We should. Why don’t we? The answer is simple and not one that anyone wants to hear. A choice has been made to keep our hosting infrastructure in it’s current state, a vulnerable state, until we secure additional capital to improve it. We should have more humans and more redundancy. Both cost money and lots of it.

You’re probably wondering what it is we *are* doing. I’ll give you an example. Yesterday I posted a long overdue change log of what the development team has been doing over the past few months. In one of those many bullet points, I mentioned that we had flipped our database keys - primary keys and foreign keys, from a 128 byte UUID to a basic integer. One of our favorite customers emailed me and said:

Impressive! Any quick tips you can share for low impact DB maintenance?

It’s quite an unsettling feeling that we are now boasting about DB maintenance fu when we just had a ridiculous amount of downtime. Well, this is actually a great example of what we have been doing to improve and how we sacrifice our time in extra effort to avoid any time offline. It’s interesting, too, and a subject of a blog post that has never surfaced. So, I’ll just describe briefly what we did and how we did it.

Before I get started, I should explain that the reason we were forced to make this drastic change to our database schema was another fault of our own. When I first created the schema for CheddarGetter’s primary database, I decided to give UUIDs a whirl. I used them for primary keys and foreign keys. The columns are char(36). They’re hex UUIDs with dashes — the kind you get from MySQL’s UUID() function. That’s 128 bytes UTF-8. Yeah, UTF-8, entirely unecessary for these columns. Integers are far more efficient in BTREE indexes than are 128 byte strings and far smaller, too. The difference in efficiency was causing severe headaches for everyone — our dev and support teams and for our customers. We absolutely had to make this change.

The size of our database at the time was around 12G. That’s not huge but it’s large enough that making the changes we needed to make would cause hours of planned downtime if we were to use traditional means. Since outages are unacceptable, even those for planned maintence, we figured out a way to do it without affecting live services at all, not even for a minute. Instead of a few days of work, it took more than the entire month of January. We put in the extra time in order to prevent hours, likely >10 hours of database locks. Here’s how we did it:

1. Plan the new schema
   • New id columns
   • New primary keys
   • New foreign keys
2. Override database access in the app framework so the app is tolerant of changes
   • Make inserts create both keys
   • Make updates reconcile both keys
   • Make lookups by primary key work for either key
3. Make the changes

Normally when you make schema changes, you simply use an ALTER statement. The trouble with ALTERs is that the table is locked while the alter takes place. When an ALTER is executed, the entire table must be recreated. For a single 4G table, for example, that can take a long time, maybe an hour or more. Ours was a unique situation because we were going to have to run an ALTER more than once for the same table because we were dealing with primary and foreign keys. In short, these changes cascade into other tables and had to be well orchestrated. Each table had to be altered, then each related table had to be altered, then the original table altered again, and the related tables again, too. That’s craziness, so we found a way to do it without any downtime.

Enter Percona Toolkit’s pt-online-schema-change, a well-named tool. An ALTER using pt-online-schema-change 1) creates a copy of the target table schema, 2) runs the ALTER on the new empty table, 3) copies all the rows from the old table, 4) ensures the two tables are in sync, 5) replaces the old table with the new. There’s a lot of database trigger witchcraft mixed in there and it works quite well. The sacrifice is time. It takes around 10x longer to alter a table using pt-online-schema-change than a traditional alter.

Here’s a simple example of what we needed to do and how we did it with pt-online-schema-change:

Consider this simple schema:

CREATE TABLE Caves (
	id char(36) not null primary key,
	location varchar(255),
	createdDatetime datetime
) ENGINE=InnoDB;

CREATE TABLE Bears (
	id char(36) not null primary key,
	caveId char(36) not null,
	constraint caveId_fk foreign key (caveId) 
		references Caves(id),
	name varchar(255),
	createdDatetime datetime
) ENGINE=InnoDB;	

First, alter the table remove the primary key, add the new primary and set an index on the old primary:

alter table Caves 
	add unique index id(id), 
	drop primary key, 
	add column newId int unsigned 
		not null primary key auto_increment FIRST;

… and the pt-online-schema-change equivalent (two steps required):

pt-online-schema-change --alter \
"add column newId int unsigned not null \
auto_increment FIRST, add index newId(newId)" \
D=testdb,t=Caves --lock-wait-time=50 \
--alter-foreign-keys-method auto \
--nocheck-replication-filters --dry-run

pt-online-schema-change --alter \
"drop primary key, drop index newId, \
add primary key(newId), add unique index id(id)" \
D=testdb,t=Caves --lock-wait-time=50 \
--alter-foreign-keys-method auto \
--nocheck-replication-filters --dry-run

Next, do the same for the related table(s) but also make foreign key change(s):

alter table Bears 
	add unique index id(id), 
	drop primary key, 
	add column newId int unsigned 
		not null primary key auto_increment FIRST,
	add column caveNewId int unsigned not null after id;

pt-online-schema-change --alter \
"add column newId int unsigned not null \
auto_increment FIRST, add index newId(newId)" \
D=testdb,t=Bears --lock-wait-time=50 \
--alter-foreign-keys-method auto \
--nocheck-replication-filters --dry-run

pt-online-schema-change --alter \
"drop primary key, drop index newId, \
add primary key(newId), add unique index id(id), \
add column caveNewId int unsigned after id" \
D=testdb,t=Bears --lock-wait-time=50 \
--alter-foreign-keys-method auto \
--nocheck-replication-filters --dry-run

Populate the new foreign key:

update Bears b, Caves c 
	set b.caveNewId=c.newId 
	where b.caveId=c.id;

We did this a block at a time using the `watch` command:

watch -n 0.1 'mysql -e "select count(*) from Bears \
where caveNewId is null; update Bears b \
set b.newCaveId=(select b.newId from Bears b \
where b.caveId=c.id) \
where b.newCaveId is null limit 50000;" testdb'

Last, setup the new foreign key(s):

alter table Bears 
	add constraint cave_fk foreign key (caveNewid) \
	references Caves(newId);

pt-online-schema-change --alter \
"cave_fk foreign key (caveNewid) \
references Caves(newId)" D=testdb,t=Bears \
--lock-wait-time=50 --alter-foreign-keys-method auto \
--nocheck-replication-filters --dry-run

For one of our tables, that process took 6 days.

Alright, long story short, we’re working hard to improve and to avoid as much downtime as possible. Things are going to get better. Honest.

Integrate CheddarGetter With Anything, Using Zapier!

Zapier is a sort of universal adapter for APIs. We have worked with their team to integrate CheddarGetter’s API in the most comprehensive way possible. This means that you can use Zapier to connect your CheddarGetter account to a ton of other services across the web.

If you want to try out some “zaps”, sign up for an account using our coupon code and you can get a 50% discount if you decide to upgrade to their paid plan. It’s free to start.

→ Try Zapier using code: cheddargetter_ef223

Zapier Integration Recipes

Here are a few example recipes using some of the services we regularly use at CheddarGetter:

1. New Subscription to SMS
2. New Subscription to MailChimp
3. New Subscription Email Draft
4. New Subscription to Basecamp Message
5. New Subscription to Highrise Contact
6. New Transaction to Ambassador

If you don’t see what you need, you can make your own “Zaps” very easily with their simple editor.

Try some of these combinations:

1. CheddarGetter with Freshbooks
2. CheddarGetter with Quickbooks
3. CheddarGetter with Zendesk
4. CheddarGetter with Magento
5. CheddarGetter with Salesforce

Zapier Is Open

The best thing about integrating with Zapier? It’s completely open. This means that any service you happen to use is a potential integration for CheddarGetter. You can scan their giant list of companies that are already integrated with Zapier, and if you find something you like, create a recipe between CheddarGetter and the system of your choosing.

See the list of currently possible integrations here →

Open also means that if your service is not on that list, you can add it yourself. That requires a bit of coding on your part, but if it’s something you think is worthwhile, chances are someone else will too. Once you write an integration, it’s available for all other Zapier users. Good karma all around.

System Performance: Database Server Upgrade

As of Sunday 2013-01-13, we’re living on a new, higher-powered master database server.

We’ve been planning to make this move for some time. Our database had grown too large to hold in RAM on our previous server, and some queries were taking upwards of a minute to complete. That was unacceptable, so we began the process of building out a bigger server, with architecture improvements to allow us to scale more easily. The new server had been setup for about 6 days and was happily replicating our production data in real time. We had run almost all of our pre-migration tests, and we were planning to make the switch this week.

In the wee hours of Sunday morning EST, our hand was forced. Our old database server failed, and we were compelled to make the switch to the new server early. There were a couple glitches during the switchover that caused the database to be down longer than it should have been. For that we’re very sorry. I mentioned that we had run *almost* all of our tests. It turns out that the ones we hadn’t gotten to would have shown us a potential problem earlier.

In the end though, the cutover went smoothly, and we’re now up and running full steam on a vastly improved master server.

On the agenda this year is a project to improve our database layer. Today we have a single point of failure at the database. That’s obviously not ideal. Worst case scenario, we could again experience unplanned downtime of a couple of hours or more. But the potential downtime grows as the size of the database grows. We maintain at all times verified real-time replicated data on a passive slave to be ready for such a catastrophe. Our current architecture requires that human interaction take place to fail over to the slave server. We run tests to determine if there is data loss and point the app to the new database host. That process can take some time.

With the database layer improvement project, we will gain the benefit of increased hardware resources and an architecture that ideally eliminates the possibility of downtime caused by failure of a single database node. In other words, the database will no longer be a single point of failure. We’ll have more on this later.

2012 Completed Projects

Here is a list of our primary development projects from last year, excluding some minor updates and fixes. If you have questions about anything here, just hit us up in Support!

• Credits, Refunds, and Voids, oh my
• Webhooks
• Promotions and Coupons
• Subscription Billable hook
• Invoice Review audit process
• Apple in-app purchase subscription conversion
• Add support for the Beanstream gateway
• API response caching - Big speed and bandwidth improvements!
• Add sales tax information to hosted pages.
• Resend payment, decline, void, refund emails
• More background processing = performance enhancements
• One-time invoice total amount can equal zero
• Improve gateway decline/error response handling
• Improve support system integration
• Remote IP passthru to all gateways
• Search customers by coupon code
• Add marketing metrics in email templates
• Enable add-charge, edit-charge, delete-charge, set-item-quantity GUI and API for outstanding invoice.
• Add waive and run invoice actions for outstanding invoice. Supports autonomous reactivation.
• Opcode caching
• Improve revenue report
• Interactive integration guide.
• PayPal error response improvements
• PayPal WPP undocumented response codes handled
• Fix PayPal standard reactivation after cancellation
• Fix PayPal preapproval limit currency conversion
• Enable PayPal subscription method in admin GUI.
• PayPal third-party permission support — improved preapproval workflow.
• Use memo field in PayPal preapproval to inform customer of subscribed plan information

Whew.

Here is the full changelog.

Recursion For Developers

All Developer accounts will soon be enabled with CheddarGetter’s production style recursion system. If you are already a paying CheddarGetter customer, you don’t need to do anything, or even read any more of this. You will not be affected.

This means that pretty much everything will behave exactly as it will when you go live. Test away!

Monday, 10/22, at 12pm noon EST is when this will happen.

Three things you should know…

1. Test Customer Behavior
   If a test customer has been billed before, they will be billed on the next invoice date. So if you created a customer on June 25th and set them on a monthly plan, a new billing datetime will be set for October 25th and the invoice will be transacted on that date. If a test customer has never been billed, they will behave as if they signed up Monday 10/22 at 12pm EST.
2. Emails Are Disabled
   Many developers have multiple test customers, and we don’t want you to get hit by a rogue wave of automated customer communication. We’ve disabled emails on all dev accounts, but you can switch them back on here:
   Turn On Customer Emails →
3. You Can Start Fresh
   Speaking of multiple test customers, you may want to start fresh with a blank slate. You can check and see if you have any test customers here:
   View Your Customers →
   And if you want to delete them all in one fell swoop, you can do that here:
   Delete Customers →
   Note: delete all only works in Developer mode (for obvious reasons)